Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71647

Add option to prevent git fetch for freestyle projects

      In our Jenkins build server we have a number of freestyle build jobs. Those build jobs are all working in the same workspace and each of those build jobs is calling another one after being finished: A->B->C->D->.... which is very time consuming. All build jobs have been configured for the same git repository to be able to track which developers contributed to the build and to checkout the same commit in each build. ... Some of those build jobs should be able to run in parallel since they only execute unit tests in separate subdirectories.

      To minimize execution time of the whole build chain we wanted to execute build jobs B, C and D in parallel: A->[B,C,D,...].

      Unfortunately, when B,C and D are started at the same time, they are all trying to fetch from the origin repository via git and store the update in the very same workspace's .git directory, which in the end results in failed builds for B, C and D.

      The error message in the build looks similar to this in such cases:

      ...
      [new branch] feature -> origin/feature
      error: cannot lock ref 'refs/remotes/origin/feature/task': is at d91b4e33b4a2d3404c0738c85b39314bb1d3c2be but expected 1b9026360df650a3ae1e224e09e92373fcafd9ef

      I think a workaround would be here to add an option in the git configuration to prevent polling of the git repository at all. Indeed, this option is already available for pipeline build jobs, but for freestyle ones it is missing.

      Can we have this option implemented also for freestyle build jobs?

      https://www.jenkins.io/doc/pipeline/steps/git/ - Search for "Example: Git step with https protocol and polling disabled"

          [JENKINS-71647] Add option to prevent git fetch for freestyle projects

          Mark Waite added a comment -

          I don't plan to implement this feature.

          The message "cannot lock ref" is usually an indication that more than one job is attempting to use the same directory for its Jenkins workspace. If that is what is happening in your case, then you'll have unexpected results as the builds collide with each other in unexpected ways. Let Jenkins determine the workspace so that it avoids the collisions for you.

          Mark Waite added a comment - I don't plan to implement this feature. The message "cannot lock ref" is usually an indication that more than one job is attempting to use the same directory for its Jenkins workspace. If that is what is happening in your case, then you'll have unexpected results as the builds collide with each other in unexpected ways. Let Jenkins determine the workspace so that it avoids the collisions for you.

          Uwe Scholz added a comment -

          Yes, indeed the build jobs all share the same workspace because of a reason: Build job A is fetching the initial sources from git (after a build was triggered externally) and it builds the executables for the unit tests, which are executed in B, C, D and so on. Since we are working on a very big mono-repo it would be a huge waste of disk space to give each unit test build job an own workspace. (in our case the repo size is like >2GB).

          Apart from that, I wonder why the option to prevent polling is available for descriptive pipeline jobs, but not for freestyle jobs. Wouldn't it be good to align their options?

          Uwe Scholz added a comment - Yes, indeed the build jobs all share the same workspace because of a reason: Build job A is fetching the initial sources from git (after a build was triggered externally) and it builds the executables for the unit tests, which are executed in B, C, D and so on. Since we are working on a very big mono-repo it would be a huge waste of disk space to give each unit test build job an own workspace. (in our case the repo size is like >2GB). Apart from that, I wonder why the option to prevent polling is available for descriptive pipeline jobs, but not for freestyle jobs. Wouldn't it be good to align their options?

          Mark Waite added a comment -

          I believe you will have undesirable surprises that are difficult to diagnose due to your choice to have multiple concurrent jobs share the same Jenkins workspace.

          I wonder why the option to prevent polling is available for descriptive pipeline jobs, but not for freestyle jobs. Wouldn't it be good to align their options?

          Pipeline polling is controlled at a level above the git plugin.

          Mark Waite added a comment - I believe you will have undesirable surprises that are difficult to diagnose due to your choice to have multiple concurrent jobs share the same Jenkins workspace. I wonder why the option to prevent polling is available for descriptive pipeline jobs, but not for freestyle jobs. Wouldn't it be good to align their options? Pipeline polling is controlled at a level above the git plugin.

          Mark Waite added a comment - - edited

          You might compare the cost of diagnosis with the cost of additional disc space.

          Alternatively, you might consider improving the performance of workspace creation by using a reference repository that keeps a copy of the large monorepo on the agent and uses the reference repository feature to only copy content that is not already in the reference repository. I created a video that describes reference repositories and other performance improvement tricks that I learned while wrestling with a 20+ GB repository.

          Mark Waite added a comment - - edited You might compare the cost of diagnosis with the cost of additional disc space. Alternatively, you might consider improving the performance of workspace creation by using a reference repository that keeps a copy of the large monorepo on the agent and uses the reference repository feature to only copy content that is not already in the reference repository. I created a video that describes reference repositories and other performance improvement tricks that I learned while wrestling with a 20+ GB repository.

          Uwe Scholz added a comment -

          Thanks for the suggestion with reference repos. I stumbled over it yesterday already. It turned out that it would not be the ideal solution here since our Jenkins nodes are created dynamically in a cloud environment and their discs are also assigned dynamically. We would have to create a reference repository for every dynamically created Jenkins node by some scripts before the initial build (in build job A). This would be beyond the scope here.

          Probably we will try to migrate our build jobs over to pipeline scripts then. At the moment I don't see a reason why the parallel unit test execution in subdirectories should influence each other. Do you foresee any possible problems I might not know?

          Uwe Scholz added a comment - Thanks for the suggestion with reference repos. I stumbled over it yesterday already. It turned out that it would not be the ideal solution here since our Jenkins nodes are created dynamically in a cloud environment and their discs are also assigned dynamically. We would have to create a reference repository for every dynamically created Jenkins node by some scripts before the initial build (in build job A). This would be beyond the scope here. Probably we will try to migrate our build jobs over to pipeline scripts then. At the moment I don't see a reason why the parallel unit test execution in subdirectories should influence each other. Do you foresee any possible problems I might not know?

          Mark Waite added a comment -

          Since you provision new nodes dynamically, why not define the base operating system image of the new node to include the reference repository? The reference repository becomes a standard part of the agent and each time the base operating system image is updated (new patches, new features, etc.), that gives an opportunity to refresh the reference repository. That shifts a large portion of the 2 GB repository data transfer into the virtual machine cloning process (often using a local disc drive with deduplicated local file system) instead of performing a 2 GB repository data transfer over the network.

          Including the reference repository will dramatically reduce the data transfer for initial clone. That reduces the startup cost of the test environment and provides faster feedback to the people that want to know the results of the tests.

          I don't see a reason why the parallel unit test execution in subdirectories should influence each other. Do you foresee any possible problems I might not know?

          Since most of the developers probably don't run parallel tests in multiple subdirectories at the same time, they don't detect the assumptions they have made that cause tests to collide. Whether those collisions are at the file system, the network, or some other resource, the parallel test environment will be checking test configurations that the developers have not checked. When test environments check things, they tend to find things. In this case, I think they will find things that the developers will probably declare to be irrelevant and artifacts of the test configuration, not a real world configuration.

          Mark Waite added a comment - Since you provision new nodes dynamically, why not define the base operating system image of the new node to include the reference repository? The reference repository becomes a standard part of the agent and each time the base operating system image is updated (new patches, new features, etc.), that gives an opportunity to refresh the reference repository. That shifts a large portion of the 2 GB repository data transfer into the virtual machine cloning process (often using a local disc drive with deduplicated local file system) instead of performing a 2 GB repository data transfer over the network. Including the reference repository will dramatically reduce the data transfer for initial clone. That reduces the startup cost of the test environment and provides faster feedback to the people that want to know the results of the tests. I don't see a reason why the parallel unit test execution in subdirectories should influence each other. Do you foresee any possible problems I might not know? Since most of the developers probably don't run parallel tests in multiple subdirectories at the same time, they don't detect the assumptions they have made that cause tests to collide. Whether those collisions are at the file system, the network, or some other resource, the parallel test environment will be checking test configurations that the developers have not checked. When test environments check things, they tend to find things. In this case, I think they will find things that the developers will probably declare to be irrelevant and artifacts of the test configuration, not a real world configuration.

          Uwe Scholz added a comment - - edited

          Oh, and one more thing: We are using the "Jenkins Parameterized Trigger plugin" to "Pass-through Git Commit that was built" from one build job to another: Each build job in our chain does checkout the exact same commit from the fetched repo. Here I don't see why it should be necessary to poll the remove repository in every build job after job A. The git repository in the workspace should contain everything already, no poll should be needed. That's why I think a checkbox to prevent the polling would be a good thing.

          Uwe Scholz added a comment - - edited Oh, and one more thing: We are using the " Jenkins Parameterized Trigger plugin " to "Pass-through Git Commit that was built" from one build job to another: Each build job in our chain does checkout the exact same commit from the fetched repo. Here I don't see why it should be necessary to poll the remove repository in every build job after job A. The git repository in the workspace should contain everything already, no poll should be needed. That's why I think a checkbox to prevent the polling would be a good thing.

          Uwe Scholz added a comment -

          Since you provision new nodes dynamically, why not define the base operating system image of the new node to include the reference repository? [...]

          I will investigate this, thanks for the idea.

          Uwe Scholz added a comment - Since you provision new nodes dynamically, why not define the base operating system image of the new node to include the reference repository? [...] I will investigate this, thanks for the idea.

          Mark Waite added a comment -

          Won't be implemented.

          Mark Waite added a comment - Won't be implemented.

            Unassigned Unassigned
            turboscholz Uwe Scholz
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: