Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58349

Initial lightweight checkout of Jenkinsfile much slower on huge repos

      I have a pipeline projects that runs pipeline script from Git repo.

      The repo became big, and the clone operation times out after 10 minutes.

      I added "Advances checkout behaviours" and "Advances clone behaviours" with much bigger timeout, but it still times out after 10 minutes.

       

       
      hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/master:refs/remotes/origin/master --prune" returned status code 128:
      stdout:
      stderr: remote: Enumerating objects: 6828, done.
      remote: Counting objects: 0% (1/6828)
      remote: Counting objects: 1% (69/6828)
      ...

      Receiving objects: 24% (5538/22691), 8.95 MiB | 14.00 KiB/s
      fatal: The remote end hung up unexpectedly
      fatal: early EOF
      fatal: index-pack failed

      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442)
      at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:351)
      at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:198)
      at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:174)
      at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
      at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:293)
      at hudson.model.ResourceController.execute(ResourceController.java:97)
      at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE

          [JENKINS-58349] Initial lightweight checkout of Jenkinsfile much slower on huge repos

          Mark Waite added a comment -

          Refer to the JENKINS-38973 workaround comment for this problem.

          Mark Waite added a comment - Refer to the JENKINS-38973 workaround comment for this problem.

          Oren Shpigel added a comment -

          Hi, in my case I use a standard pipeline, and not a Multibranch-Pipeline.

          I tried this workaround but it didn't change the 10 minute timeout.

          (I have Jenkins installed on ubuntu 16.04, I edited the /etc/default/jenkins JENKINS_ARGS value to have the -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=25 extra argument)

          Oren Shpigel added a comment - Hi, in my case I use a standard pipeline, and not a Multibranch-Pipeline. I tried this workaround but it didn't change the 10 minute timeout. (I have Jenkins installed on ubuntu 16.04, I edited the /etc/default/jenkins JENKINS_ARGS value to have the -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=25 extra argument)

          Mark Waite added a comment - - edited

          I suspect it is the same issue as the duplicate bug, failure to honor the timeout when reading the initial Jenkinsfile.

          I created a Pipeline definition test which confirms that the `checkout` step in the Pipeline honors the timeout. In that test, I clone a fresh copy of the linux-stable repository (1.5 GB) from a small, local, relatively low performance git server that I maintain inside my network. The Pipeline job asserts that the expected timeout error is reported during the checkout and that the `timeout=1` is correctly noted on the attempt to clone the linux-stable repository.

          I've updated the title of JENKINS-38973 to note that it covers any Pipeline initial checkout of the Jenkinsfile from a large repository.

          Further exploring seems to indicate that if I create the solo Pipeline job with the `lightweight checkout` disabled, then the timeout is honored for the initial checkout of the Jenkinsfile. The lightweight checkout enabled job takes much longer to perform the initial checkout of the Jenkinsfile (likely due to the large size of the repository which must be initially cloned for the lightweight checkout of just the Jenkinsfile).

          See the job definitions (same job, one with lightweight checkout, one without lightweight checkout) as:

          Mark Waite added a comment - - edited I suspect it is the same issue as the duplicate bug, failure to honor the timeout when reading the initial Jenkinsfile. I created a Pipeline definition test which confirms that the `checkout` step in the Pipeline honors the timeout. In that test, I clone a fresh copy of the linux-stable repository (1.5 GB) from a small, local, relatively low performance git server that I maintain inside my network. The Pipeline job asserts that the expected timeout error is reported during the checkout and that the `timeout=1` is correctly noted on the attempt to clone the linux-stable repository. I've updated the title of JENKINS-38973 to note that it covers any Pipeline initial checkout of the Jenkinsfile from a large repository. Further exploring seems to indicate that if I create the solo Pipeline job with the `lightweight checkout` disabled, then the timeout is honored for the initial checkout of the Jenkinsfile. The lightweight checkout enabled job takes much longer to perform the initial checkout of the Jenkinsfile (likely due to the large size of the repository which must be initially cloned for the lightweight checkout of just the Jenkinsfile). See the job definitions (same job, one with lightweight checkout, one without lightweight checkout) as: Without lightweight checkout - job completes in 95 seconds With lightweight checkout - job completes in 550 seconds

          Oren Shpigel added a comment -

          Disabling the "lightweight checkout" actually solves my problem, and made the timeout from the workaround to be effective.

          Thanks!!

           

          Oren Shpigel added a comment - Disabling the "lightweight checkout" actually solves my problem, and made the timeout from the workaround to be effective. Thanks!!  

            Unassigned Unassigned
            oren_icx Oren Shpigel
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: