Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70012

Perform lightweight checkout on tags

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • atlassian-bitbucket-server-integration-plugin
    • None

      It seems like Bitbucket branches follow this convention but not Bitbucket tags. I create a Multibranch Pipeline job with the Branch Source set to Bitbucket Server. I have Discover Branches and Discover Tags set as well as the Build Strategies for building all regular branches and all tags regardless of age. 

      If I look in /var/lib/jenkins on the Jenkins Built-in node after a branch job runs I see a workspace_libs/ directory with the shared libraries that job imports cloned down. However, on tag jobs, I see that workspace_libs/ folder as well as a workspace_script/ folder. That workspace_script/ folder has the entire repo cloned down in order to access the Jenkinsfile it seems.

      To cut down on clone size I have changed my job configuration to sparse checkout just the Jenkinsfile but I still get the full .git/ folder under workspace_script/ which for some of our repos can be pretty big (~380 MB - it’s a large C++ project so git-lfs won’t help us here). I can change the clone depth as well but that has the consequences of not populating the “changes” field correctly if the shallow history depth of the clone is smaller than the number of commits since the last job run (not the end of the world there but not ideal). 

      For what it’s worth, our pipelines have their own stage in a node{} block that perform their own clone on the builder node the job chooses so there’s no need for us to use that workspace_scripts/ folder. We cleanup on the builder nodes after the jobs run so that’s not a problem. The Built-in node is where the growth seems to happen that we can’t control as well as I’d hope. 

      I would imagine branches and tags should work the same right? Branches don’t seem to need that workspace_script/ folder at all in order to access and use the Jenkinsfile. Is this a bug in how the tag jobs are processed or is there a reason behind that difference in behavior?

      Thanks! Im on mobile now so can’t provide server details or screenshots but will do so first thing tomorrow morning. 

      Note: workspace_libs/ and workspace_scripts/ actually have a double underscore in their name but formatting issues here preventing me from typing that out. I called them out above with the single underscore but just know that the server shows double as expected.

          [JENKINS-70012] Perform lightweight checkout on tags

          Michael Steppe added a comment -

          Added environment details to ticket.

          Michael Steppe added a comment - Added environment details to ticket.

          Michael Steppe added a comment -

          Another datapoint:

          As stated above, the workspace_libs/ folder for both branches and tags contains a full git clone of the shared libraries (including the .git folder, files like our shared library's Jenkinsfile and README.md, etc.). I can change that depth to shrink the .git folder size but it still does a full clone.

          However, if I dive further into a branch or tag, each individual build also has a "libs/" folder with only the "resources/", "src/", and "vars/" folders from the shared library cloned down and no .git folder. That's much smaller footprint - although with many many builds, this could grow pretty quickly. Is there any reason the individual builds can't reuse the workspace_libs/ shared library clones or vice versa?

          I'm sure I'm misunderstanding something here or missing a configuration step because it seems like there's so much duplication saved on the Built-in node between access each individual build of each job's Jenkinsfile and shared libraries it requires to run.

          Michael Steppe added a comment - Another datapoint: As stated above, the workspace_libs/ folder for both branches and tags contains a full git clone of the shared libraries (including the .git folder, files like our shared library's Jenkinsfile and README.md, etc.). I can change that depth to shrink the .git folder size but it still does a full clone. However, if I dive further into a branch or tag,  each individual build also has a "libs/" folder with  only the "resources/", "src/", and "vars/" folders from the shared library cloned down and no .git folder. That's much smaller footprint - although with many many builds, this could grow pretty quickly. Is there any reason the individual builds can't reuse the workspace_libs/ shared library clones or vice versa? I'm sure I'm misunderstanding something here or missing a configuration step because it seems like there's so much duplication saved on the Built-in node between access each individual build of each job's Jenkinsfile and shared libraries it requires to run.

          Michael Steppe added a comment -

          Datapoint #2 (probably the most helpful so far):

          Neither the "Git" nor the "Bitbucket" (not Bitbucket Server) branch sources create that "workspace_script/" folder on the Built-in node for branches or tags.

          Sorry for spamming comments, I only mean to provide more data for when someone has a chance to look at this - not trolling for visibility or anything!

          Michael Steppe added a comment - Datapoint #2 (probably the most helpful so far): Neither the "Git" nor the "Bitbucket" (not Bitbucket Server) branch sources create that "workspace_script/" folder on the Built-in node for branches or tags. Sorry for spamming comments, I only mean to provide more data for when someone has a chance to look at this - not trolling for visibility or anything!

          Michael Steppe added a comment - - edited

          Not sure if this helps, but I found the config diff change that seemed to cause the ".git" folder to be saved to that workspace_scripts folder (back then it was actually called "workspace@scripts". A Bitbucket tag from before this change only had the Jenkinsfile cloned down into that folder. Tags after this change have the Jenkinsfile in addition to the .git folder. As stated in the ticket description, for large projects, that .git folder can be very big.

            Older Change
          Date:2021-04-09_09-15-57
          Operation:Changed
          User:SYSTEM          		   Newer Change
          Date:2021-10-21_05-04-30
          Operation:Changed
          User:SYSTEM
          1 <?xml version="1.0" encoding="UTF-8"?><org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectplugin="workflow-multibranch@2.23"> 1 <?xml version="1.0" encoding="UTF-8"?><org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectplugin="workflow-multibranch@2.26">
          2 <actions/> 2 <actions/>
          3 <description/> 3 <description/>
          4 <disabled>false</disabled> 4 <disabled>false</disabled>
          6 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 6 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/>
          7 <scriptPath>REDACTED</scriptPath> 7 <scriptPath>REDACTED</scriptPath>
          8 </factory> 8 </factory>
          9 <folderViewsclass="jenkins.branch.MultiBranchProjectViewHolder"plugin="branch-api@2.6.3"> 9 <folderViewsclass="jenkins.branch.MultiBranchProjectViewHolder"plugin="branch-api@2.7.0">
          10 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 10 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/>
          11 </folderViews> 11 </folderViews>
          12 <healthMetrics> 12 <healthMetrics>
          13 <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetricplugin="cloudbees-folder@6.15"> 13 <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetricplugin="cloudbees-folder@6.16">
          14 <nonRecursive>false</nonRecursive> 14 <nonRecursive>false</nonRecursive>
          15 </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric> 15 </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric>
          16 </healthMetrics> 16 </healthMetrics>
          17 <iconclass="jenkins.branch.MetadataActionFolderIcon"plugin="branch-api@2.6.3"> 17 <iconclass="jenkins.branch.MetadataActionFolderIcon"plugin="branch-api@2.7.0">
          18 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 18 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/>
          19 </icon> 19 </icon>
          20 <orphanedItemStrategyclass="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy"plugin="cloudbees-folder@6.15"> 20 <orphanedItemStrategyclass="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy"plugin="cloudbees-folder@6.16">
          21 <daysToKeep>30</daysToKeep> 21 <daysToKeep>30</daysToKeep>
          22 <numToKeep>-1</numToKeep> 22 <numToKeep>-1</numToKeep>
          23 <pruneDeadBranches>true</pruneDeadBranches> 23 <pruneDeadBranches>true</pruneDeadBranches>
          ...   ...  
          27 <dockerLabel/> 27 <dockerLabel/>
          28 <registryplugin="docker-commons@1.17"/> 28 <registryplugin="docker-commons@1.17"/>
          29 </org.jenkinsci.plugins.docker.workflow.declarative.FolderConfig> 29 </org.jenkinsci.plugins.docker.workflow.declarative.FolderConfig>
          30 <org.jenkinsci.plugins.workflow.libs.FolderLibrariesplugin="workflow-cps-global-lib@2.18"> 30 <org.jenkinsci.plugins.workflow.libs.FolderLibrariesplugin="workflow-cps-global-lib@2.21">
          31 <libraries> 31 <libraries>
          32 <org.jenkinsci.plugins.workflow.libs.LibraryConfiguration> 32 <org.jenkinsci.plugins.workflow.libs.LibraryConfiguration>
          33 <allowVersionOverride>true</allowVersionOverride> 33 <allowVersionOverride>true</allowVersionOverride>
          ...   ...  
          37 </libraries> 37 </libraries>
          38 </org.jenkinsci.plugins.workflow.libs.FolderLibraries> 38 </org.jenkinsci.plugins.workflow.libs.FolderLibraries>
          39 </properties> 39 </properties>
          40 <sourcesclass="jenkins.branch.MultiBranchProject$BranchSourceList"plugin="branch-api@2.6.3"> 40 <sourcesclass="jenkins.branch.MultiBranchProject$BranchSourceList"plugin="branch-api@2.7.0">
          41 <data> 41 <data>
          42 <jenkins.branch.BranchSource> 42 <jenkins.branch.BranchSource>
          43 <sourceclass="com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCMSource"plugin="atlassian-bitbucket-server-integration@2.1.3"> 43 <sourceclass="com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCMSource"plugin="atlassian-bitbucket-server-integration@3.0.1">
          44 <gitSCMSource> 44 <gitSCMSource>
          45 <credentialsId/> 45 <credentialsId/>
          46 <remote>REDACTED</remote> 46 <remote>REDACTED</remote>
          ...   ...  
          62 <sshCredentialsId/> 62 <sshCredentialsId/>
          63 </repository> 63 </repository>
          64 <traits> 64 <traits>
          65 <jenkins.plugins.git.traits.BranchDiscoveryTraitplugin="git@4.7.1"/> 65 <jenkins.plugins.git.traits.BranchDiscoveryTraitplugin="git@4.8.2"/>
          66 <jenkins.plugins.git.traits.SparseCheckoutPathsTraitplugin="git@4.7.1"> 66 <jenkins.plugins.git.traits.SparseCheckoutPathsTraitplugin="git@4.8.2">
          67 <extensionclass="hudson.plugins.git.extensions.impl.SparseCheckoutPaths"> 67 <extensionclass="hudson.plugins.git.extensions.impl.SparseCheckoutPaths">
          68 <sparseCheckoutPaths> 68 <sparseCheckoutPaths>
          69 <hudson.plugins.git.extensions.impl.SparseCheckoutPath> 69 <hudson.plugins.git.extensions.impl.SparseCheckoutPath>
          ...   ...  
          72 </sparseCheckoutPaths> 72 </sparseCheckoutPaths>
          73 </extension> 73 </extension>
          74 </jenkins.plugins.git.traits.SparseCheckoutPathsTrait> 74 </jenkins.plugins.git.traits.SparseCheckoutPathsTrait>
          75 <jenkins.plugins.git.traits.TagDiscoveryTraitplugin="git@4.7.1"/> 75 <jenkins.plugins.git.traits.TagDiscoveryTraitplugin="git@4.8.2"/>
          76 </traits> 76 </traits>
          77 <webhookRegistered>true</webhookRegistered> 77 <webhookRegistered>true</webhookRegistered>
          78 </source> 78 </source>
          ...   ...  
          88 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 88 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/>
          89 </sources> 89 </sources>
          90 <triggers> 90 <triggers>
          91 <com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTriggerplugin="atlassian-bitbucket-server-integration@2.1.3"> 91 <com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTriggerplugin="atlassian-bitbucket-server-integration@3.0.1">
              92 <pullRequestTrigger>false</pullRequestTrigger>
              93 <refTrigger>false</refTrigger>
          92 <spec/> 94 <spec/>
              95 <version>0</version>
          93 </com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTrigger> 96 </com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTrigger>
          94 <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTriggerplugin="cloudbees-folder@6.15"> 97 <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTriggerplugin="cloudbees-folder@6.16">
          95 <interval>604800000</interval> 98 <interval>604800000</interval>
          96 <spec>H H * * *</spec> 99 <spec>H H * * *</spec>
          97 </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger> 100 </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>

          Michael Steppe added a comment - - edited Not sure if this helps, but I found the config diff change that seemed to cause the ".git" folder to be saved to that workspace_scripts folder (back then it was actually called "workspace@scripts". A Bitbucket tag from before this change only had the Jenkinsfile cloned down into that folder. Tags after this change have the Jenkinsfile in addition to the .git folder. As stated in the ticket description, for large projects, that .git folder can be very big.   Older Change Date:2021-04-09_09-15-57 Operation:Changed User:SYSTEM   Newer Change Date:2021-10-21_05-04-30 Operation:Changed User:SYSTEM 1 <?xml version="1.0" encoding="UTF-8"?><org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectplugin="workflow-multibranch@2.23"> 1 <?xml version="1.0" encoding="UTF-8"?><org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectplugin="workflow-multibranch@2.26"> 2 <actions/> 2 <actions/> 3 <description/> 3 <description/> 4 <disabled>false</disabled> 4 <disabled>false</disabled> 6 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 6 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 7 <scriptPath>REDACTED</scriptPath> 7 <scriptPath>REDACTED</scriptPath> 8 </factory> 8 </factory> 9 <folderViewsclass="jenkins.branch.MultiBranchProjectViewHolder"plugin="branch-api@2.6.3"> 9 <folderViewsclass="jenkins.branch.MultiBranchProjectViewHolder"plugin="branch-api@2.7.0"> 10 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 10 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 11 </folderViews> 11 </folderViews> 12 <healthMetrics> 12 <healthMetrics> 13 <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetricplugin="cloudbees-folder@6.15"> 13 <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetricplugin="cloudbees-folder@6.16"> 14 <nonRecursive>false</nonRecursive> 14 <nonRecursive>false</nonRecursive> 15 </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric> 15 </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric> 16 </healthMetrics> 16 </healthMetrics> 17 <iconclass="jenkins.branch.MetadataActionFolderIcon"plugin="branch-api@2.6.3"> 17 <iconclass="jenkins.branch.MetadataActionFolderIcon"plugin="branch-api@2.7.0"> 18 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 18 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 19 </icon> 19 </icon> 20 <orphanedItemStrategyclass="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy"plugin="cloudbees-folder@6.15"> 20 <orphanedItemStrategyclass="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy"plugin="cloudbees-folder@6.16"> 21 <daysToKeep>30</daysToKeep> 21 <daysToKeep>30</daysToKeep> 22 <numToKeep>-1</numToKeep> 22 <numToKeep>-1</numToKeep> 23 <pruneDeadBranches>true</pruneDeadBranches> 23 <pruneDeadBranches>true</pruneDeadBranches> ...   ...   27 <dockerLabel/> 27 <dockerLabel/> 28 <registryplugin="docker-commons@1.17"/> 28 <registryplugin="docker-commons@1.17"/> 29 </org.jenkinsci.plugins.docker.workflow.declarative.FolderConfig> 29 </org.jenkinsci.plugins.docker.workflow.declarative.FolderConfig> 30 <org.jenkinsci.plugins.workflow.libs.FolderLibrariesplugin="workflow-cps-global-lib@2.18"> 30 <org.jenkinsci.plugins.workflow.libs.FolderLibrariesplugin="workflow-cps-global-lib@2.21"> 31 <libraries> 31 <libraries> 32 <org.jenkinsci.plugins.workflow.libs.LibraryConfiguration> 32 <org.jenkinsci.plugins.workflow.libs.LibraryConfiguration> 33 <allowVersionOverride>true</allowVersionOverride> 33 <allowVersionOverride>true</allowVersionOverride> ...   ...   37 </libraries> 37 </libraries> 38 </org.jenkinsci.plugins.workflow.libs.FolderLibraries> 38 </org.jenkinsci.plugins.workflow.libs.FolderLibraries> 39 </properties> 39 </properties> 40 <sourcesclass="jenkins.branch.MultiBranchProject$BranchSourceList"plugin="branch-api@2.6.3"> 40 <sourcesclass="jenkins.branch.MultiBranchProject$BranchSourceList"plugin="branch-api@2.7.0"> 41 <data> 41 <data> 42 <jenkins.branch.BranchSource> 42 <jenkins.branch.BranchSource> 43 <sourceclass="com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCMSource"plugin="atlassian-bitbucket-server-integration@2.1.3"> 43 <sourceclass="com.atlassian.bitbucket.jenkins.internal.scm.BitbucketSCMSource"plugin="atlassian-bitbucket-server-integration@3.0.1"> 44 <gitSCMSource> 44 <gitSCMSource> 45 <credentialsId/> 45 <credentialsId/> 46 <remote>REDACTED</remote> 46 <remote>REDACTED</remote> ...   ...   62 <sshCredentialsId/> 62 <sshCredentialsId/> 63 </repository> 63 </repository> 64 <traits> 64 <traits> 65 <jenkins.plugins.git.traits.BranchDiscoveryTraitplugin="git@4.7.1"/> 65 <jenkins.plugins.git.traits.BranchDiscoveryTraitplugin="git@4.8.2"/> 66 <jenkins.plugins.git.traits.SparseCheckoutPathsTraitplugin="git@4.7.1"> 66 <jenkins.plugins.git.traits.SparseCheckoutPathsTraitplugin="git@4.8.2"> 67 <extensionclass="hudson.plugins.git.extensions.impl.SparseCheckoutPaths"> 67 <extensionclass="hudson.plugins.git.extensions.impl.SparseCheckoutPaths"> 68 <sparseCheckoutPaths> 68 <sparseCheckoutPaths> 69 <hudson.plugins.git.extensions.impl.SparseCheckoutPath> 69 <hudson.plugins.git.extensions.impl.SparseCheckoutPath> ...   ...   72 </sparseCheckoutPaths> 72 </sparseCheckoutPaths> 73 </extension> 73 </extension> 74 </jenkins.plugins.git.traits.SparseCheckoutPathsTrait> 74 </jenkins.plugins.git.traits.SparseCheckoutPathsTrait> 75 <jenkins.plugins.git.traits.TagDiscoveryTraitplugin="git@4.7.1"/> 75 <jenkins.plugins.git.traits.TagDiscoveryTraitplugin="git@4.8.2"/> 76 </traits> 76 </traits> 77 <webhookRegistered>true</webhookRegistered> 77 <webhookRegistered>true</webhookRegistered> 78 </source> 78 </source> ...   ...   88 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 88 <ownerclass="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject"reference="../.."/> 89 </sources> 89 </sources> 90 <triggers> 90 <triggers> 91 <com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTriggerplugin="atlassian-bitbucket-server-integration@2.1.3"> 91 <com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTriggerplugin="atlassian-bitbucket-server-integration@3.0.1">     92 <pullRequestTrigger>false</pullRequestTrigger>     93 <refTrigger>false</refTrigger> 92 <spec/> 94 <spec/>     95 <version>0</version> 93 </com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTrigger> 96 </com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookMultibranchTrigger> 94 <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTriggerplugin="cloudbees-folder@6.15"> 97 <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTriggerplugin="cloudbees-folder@6.16"> 95 <interval>604800000</interval> 98 <interval>604800000</interval> 96 <spec>H H * * *</spec> 99 <spec>H H * * *</spec> 97 </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger> 100 </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>

            Unassigned Unassigned
            j17359 Michael Steppe
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: