Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38973

Pipeline initial checkout of Jenkinsfile not honoring git timeout

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins 2.19.1
      Git client plugin 2.0.0
      Git plugin 3.0.0
      Pipeline: Multibranch: 2.9

      I am having an issue cloning a large repo for use in a multi-branch pipeline project.

      My goal is to run pipeline jobs against certain branches within my repository. However, the initial fetch seems to timeout. As you can see from the attached log, the issue appears to arise from a 10 minute timeout issue. However, if you see the below screen shots, I have all of my timeouts set to 180minutes.

      In the attached tickets there were known issues with the 10 minute git timeout that were fixed by allowing users access to overrides as I have shown in the attached images. It appears as though these overrides are not being honored for some reason within the MultibranchPipeline project.

      I have used the settings described here on a smaller repo and it was able to properly index the branches.

      As a work around, I also tried only using the overrides as advanced settings for checkout/fetch behavior. This did not work either, unfortunately.

          [JENKINS-38973] Pipeline initial checkout of Jenkinsfile not honoring git timeout

          We have this issue too. Always timeout on indexing.

          Erik Hakansson added a comment - We have this issue too. Always timeout on indexing.

          This pull request to Git client plugin makes the default timeout configurable and should solve this issue.

          Erik Hakansson added a comment - This pull request to Git client plugin makes the default timeout configurable and should solve this issue.

          Real Name added a comment -

          Huge blocker. Needs to be fixed.
          Any available workarounds?

          Real Name added a comment - Huge blocker. Needs to be fixed. Any available workarounds?

          Tino Siegmund added a comment -

          Same problem here. Tryed to manually pre clone the repository does not help.

          Tino Siegmund added a comment - Same problem here. Tryed to manually pre clone the repository does not help.

          Sean MacKay added a comment -

          This is suddenly an issue for me. I'm not sure the timeout is the culprit, as it used to clone fine, but now some problem with the git fetch does cause the timeout to expire.

          Additionally, this doesn't read as a failure on the git fetch for the multibranch pipeline project, it just stays locked on this git fetch process preventing other jobs from running. Perhaps this is another issue that should have a new bug opened?

          Sean MacKay added a comment - This is suddenly an issue for me. I'm not sure the timeout is the culprit, as it used to clone fine, but now some problem with the git fetch does cause the timeout to expire. Additionally, this doesn't read as a failure on the git fetch for the multibranch pipeline project, it just stays locked on this git fetch process preventing other jobs from running. Perhaps this is another issue that should have a new bug opened?

          Mark Waite added a comment -

          An alternative might be to pass -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=37 as a Java command line option to Jenkins. That will set the Jenkins global git timeout to 37 minutes instead of its default of 10 minutes.

          I don't know why the pipeline initial clone ignores the timeout setting, but I've confirmed that it does ignore it.

          I confirmed it is ignored with the following steps:

          1. Create a multi-branch pipeline job with Git as the SCM source
          2. Use https://github.com/torvalds/linux.git as the repository location
          3. Add the "Additional Behaviour" for "Advanced clone options"
          4. Set the timeout to a small value (1 or 2)
          5. Click the "Scan Multibranch Pipeline Now" link
          6. Monitor the progress of the clone, note that it will run longer than the assigned timeout value. Other cases honor the timeout, but apparently not that case

          Mark Waite added a comment - An alternative might be to pass -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=37 as a Java command line option to Jenkins. That will set the Jenkins global git timeout to 37 minutes instead of its default of 10 minutes. I don't know why the pipeline initial clone ignores the timeout setting, but I've confirmed that it does ignore it. I confirmed it is ignored with the following steps: Create a multi-branch pipeline job with Git as the SCM source Use https://github.com/torvalds/linux.git as the repository location Add the "Additional Behaviour" for "Advanced clone options" Set the timeout to a small value (1 or 2) Click the "Scan Multibranch Pipeline Now" link Monitor the progress of the clone, note that it will run longer than the assigned timeout value. Other cases honor the timeout, but apparently not that case

          hardi249 added a comment -

          -Dorg.jenkinsci.plugins.gitclient.Git.timeOut does nothing. The initial checkout/fetch for each triggered branch job still defaults to 10 minutes.

          hardi249 added a comment - -Dorg.jenkinsci.plugins.gitclient.Git.timeOut does nothing. The initial checkout/fetch for each triggered branch job still defaults to 10 minutes.

          John Lozano added a comment -

          +1 on this issue.

          Passing the org.jenkinsci.plugins.gitclient.Git.timeOut in as JAVA_OPTS, using setProperty() from the script console, and manually configuring the clone behavior on a multi-branch pipeline job all fail to modify the timeout used for the initial checkout/fetch when indexing branches. 

          I really want to try out this plugin in order to simplify my current build strategy, but I don't want to resort to splitting up my repository.

          Using TFS/Git as repository provider.

          John Lozano added a comment - +1 on this issue. Passing the org.jenkinsci.plugins.gitclient.Git.timeOut in as JAVA_OPTS, using setProperty() from the script console, and manually configuring the clone behavior on a multi-branch pipeline job all fail to modify the timeout used for the initial checkout/fetch when indexing branches.  I really want to try out this plugin in order to simplify my current build strategy, but I don't want to resort to splitting up my repository. Using TFS/Git as repository provider.

          Mark Waite added a comment -

          lozjo have you considered using a reference repository as a way to significantly reduce the clone time and the disc use of your large git repository?  Refer to my Jenkins World 2017 presentation for an overview of the options for better handling large repositories in Jenkins.

          Mark Waite added a comment - lozjo have you considered using a reference repository as a way to significantly reduce the clone time and the disc use of your large git repository?  Refer to my Jenkins World 2017 presentation for an overview of the options for better handling large repositories in Jenkins.

          +1 this issue.

          I have a local reference repository on the disk and, but it is ignored. Just like the timeout option.

          See attachments.

          would LOVE to have a fix or workaround for this. I tried digging into the code but I am a total newbie

          Wouldn't the solution be to add a configurable timeout and reference repository to the 'Branch Discovery' feature ?

           

          guillaume docquier added a comment - +1 this issue. I have a local reference repository on the disk and, but it is ignored. Just like the timeout option. See attachments. would LOVE to have a fix or workaround for this. I tried digging into the code but I am a total newbie Wouldn't the solution be to add a configurable timeout and reference repository to the 'Branch Discovery' feature ?  

          Hi, 

          g_doc2080 please check this link: https://stackoverflow.com/a/37133464 

          Antonio Asenjo added a comment - Hi,  g_doc2080  please check this link:  https://stackoverflow.com/a/37133464  

          Hi,

          I second g_doc2080's comment. Our git repository is close to 5GB in size and the local git repository is not being used at all when running "Branch Indexing".

          aseg: thanks for the link but setting the timeout option higher is not really a long term solution since we have a large number of pipelines with all require branch indexing meaning the 5GB will be duplicated many times on the master.

          I too would like to see a solution for this. I will have a look into the code when I have time but I don't know when this will be.

           

          Steve Boardwell added a comment - Hi, I second g_doc2080 's comment. Our git repository is close to 5GB in size and the local git repository is not being used at all when running "Branch Indexing". aseg : thanks for the link but setting the timeout option higher is not really a long term solution since we have a large number of pipelines with all require branch indexing meaning the 5GB will be duplicated many times on the master. I too would like to see a solution for this. I will have a look into the code when I have time but I don't know when this will be.  

          Not exactly the same, but we have a similar problem with the shallow clone option: the plugin doesn't seem to take into account either (no --depth=xx on the git fetch command line)

          Sebastien Hoareau added a comment - Not exactly the same, but we have a similar problem with the shallow clone option: the plugin doesn't seem to take into account either (no --depth=xx on the git fetch command line)

          Rohit Gagan added a comment -

          Any resolution to this issue. Still facing the same problem

          Rohit Gagan added a comment - Any resolution to this issue. Still facing the same problem

          Mark Waite added a comment - - edited

          No rohit_gagan, the best work around is still the stackoverflow article referenced by aseg. Since branch indexing happens on the master and uses a known cache, it should create only a single copy of the large repository, then it will reuse that for additional operations.

          Mark Waite added a comment - - edited No rohit_gagan , the best work around is still the stackoverflow article referenced by aseg . Since branch indexing happens on the master and uses a known cache, it should create only a single copy of the large repository, then it will reuse that for additional operations.

          Rohit Gagan added a comment -

          markewaite. Tried adding that, this still takes timeout as 10 min

           

          Started
          [Wed May 22 11:55:15 EDT 2019] Starting branch indexing...
          > git --version # timeout=10

          Rohit Gagan added a comment - markewaite . Tried adding that, this still takes timeout as 10 min   Started [Wed May 22 11:55:15 EDT 2019] Starting branch indexing... > git --version # timeout=10

          Rohit Gagan added a comment -

          markewaite only option i see is change in git plugin code then to increase the default timeout. What you suggest?

           

          Rohit Gagan added a comment - markewaite only option i see is change in git plugin code then to increase the default timeout. What you suggest?  

          Rohit Gagan added a comment -

          markewaite. Thanks. This works. i was adding the parameter at the wrong end

          Rohit Gagan added a comment - markewaite . Thanks. This works. i was adding the parameter at the wrong end

          Joel added a comment -

          I have this issue too (index-pack died of signal 15) because our repo is too big (20+GB), even though I gave lightweight checkout enabled.

          I'm wondering why Jenkins is not using git archive as an alternative to checking out the whole repository.

          The current method could still be kept as a fallback if the remote has not enabled that option (like for GitHub, for example)

          Joel added a comment - I have this issue too ( index-pack died of signal 15 ) because our repo is too big (20+GB), even though I gave lightweight checkout enabled. I'm wondering why Jenkins is not using git archive  as an alternative to checking out the whole repository. The current method could still be kept as a fallback if the remote has not enabled that option (like for GitHub , for example)

          Mark Waite added a comment -

          glatapoui refer to "Git in the Large - Tips and Tricks" for recommendations of techniques you can use to reduce the challenges of large git repositories. The techniques that have been most effective for me have included (in order of effectiveness):

          1. Use git large file support rather than storing large files directly in git
          2. Place a reference copy of the repository on the agent that is receiving the large repository and update it periodically (weekly, monthly)
          3. Narrow the refspec used in the job definition so that only the necessary branches are fetched from the remote to the local repository (can be a very significant win if there are many branches in the repository and the content of the branches is mostly distinct from other branches)
          4. Use shallow clone to limit the history depth that is retrieved (sometimes this is known to place too heavy a load on the remote server and should not be used)
          5. Use sparse checkout to reduce the disc use for the agent checkout

          I don't plan to implement git archive support in the git plugin.

          Mark Waite added a comment - glatapoui refer to "Git in the Large - Tips and Tricks" for recommendations of techniques you can use to reduce the challenges of large git repositories. The techniques that have been most effective for me have included (in order of effectiveness): Use git large file support rather than storing large files directly in git Place a reference copy of the repository on the agent that is receiving the large repository and update it periodically (weekly, monthly) Narrow the refspec used in the job definition so that only the necessary branches are fetched from the remote to the local repository (can be a very significant win if there are many branches in the repository and the content of the branches is mostly distinct from other branches) Use shallow clone to limit the history depth that is retrieved (sometimes this is known to place too heavy a load on the remote server and should not be used) Use sparse checkout to reduce the disc use for the agent checkout I don't plan to implement git archive support in the git plugin.

            Unassigned Unassigned
            magic David Johnson
            Votes:
            26 Vote for this issue
            Watchers:
            34 Start watching this issue

              Created:
              Updated: