Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20531

Retry count not working in Git plugin 2.0

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • git-plugin
    • None

    Description

      stderr: fatal: Couldn't find remote ref refs/changes/12/34567/8
      fatal: The remote end hung up unexpectedly
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:981)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:920)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.fetch(CliGitAPIImpl.java:187)
      at hudson.plugins.git.GitAPI.fetch(GitAPI.java:229)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:610)
      ... 10 more

      And there is no even attempt to retry...
      (I have local Git-mirror, which updates from Gerrit. Jenkins fetching from that mirror, so it's ok when ref not "ready" yet)
      In versions 1.* retrying works like a charm, so I have to roll back to 1.4

      Attachments

        Activity

          markewaite Mark Waite added a comment - - edited

          If the Git repository from Gerrit is mirrored to the local Git mirror and the Jenkins job is cloning from the local Git mirror, why not have the Jenkins job monitor the local git mirror?

          You could possibly use a post-receive hook for fast detection of new arrivals. That seems like coupling the Jenkins job to the local Git mirror, but launching it from a submission to an unmonitored repository, and expecting it to build a submission which may not yet have arrived in the local Git mirror is unnecessarily asymmetric.

          I'm not claiming that your report is not a bug. I think it is a bug, but it seems like you could work around that bug by monitoring the local Git repo with the Jenkins job rather than monitoring the Gerrit repo and cloning the local repo.

          Alternately, if clone performance is the issue due to a large repository, you could consider adding an "Advanced clone behaviors" to reference the local Git repo, but clone from the Gerrit repo. That advanced clone behavior has helped significantly reduce my clone times on Linux machines. Unfortunately, I have never found a way to make it work on Windows.

          markewaite Mark Waite added a comment - - edited If the Git repository from Gerrit is mirrored to the local Git mirror and the Jenkins job is cloning from the local Git mirror, why not have the Jenkins job monitor the local git mirror? You could possibly use a post-receive hook for fast detection of new arrivals. That seems like coupling the Jenkins job to the local Git mirror, but launching it from a submission to an unmonitored repository, and expecting it to build a submission which may not yet have arrived in the local Git mirror is unnecessarily asymmetric. I'm not claiming that your report is not a bug. I think it is a bug, but it seems like you could work around that bug by monitoring the local Git repo with the Jenkins job rather than monitoring the Gerrit repo and cloning the local repo. Alternately, if clone performance is the issue due to a large repository, you could consider adding an "Advanced clone behaviors" to reference the local Git repo, but clone from the Gerrit repo. That advanced clone behavior has helped significantly reduce my clone times on Linux machines. Unfortunately, I have never found a way to make it work on Windows.

          I agree, it's ugly. But there are some reasons for that, and they are beyond my control.

          sapone Sergey Mylnikov added a comment - I agree, it's ugly. But there are some reasons for that, and they are beyond my control.

          Can't support such an exotic use-case.

          ndeloof Nicolas De Loof added a comment - Can't support such an exotic use-case.

          The retry in not working for me either. I'll try to give a better analysis. I'm running Jenkins 1.554.2, git-plugin 2.2.1, git-client-plugin 1.9.1.

          A failed checkout causes a GitException as shown below. GitException is a RuntimeException. That exception is propagated all the way up to the retry handling in AbstractBuildExecution.defaultCheckout. But there only AbortException, InterruptedIOException and IOException trigger a retry.

          So the problem is that either the retry handling should catch any RuntimeException or GitSCM should not throw a RuntimeException in this case.

          hudson.plugins.git.GitException: Failed to fetch from https://github.com/foo/bar.git
          	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
          	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
          	at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
          	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
          	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
          	at hudson.model.Run.execute(Run.java:1684)
          	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          	at hudson.model.ResourceController.execute(ResourceController.java:88)
          	at hudson.model.Executor.run(Executor.java:231)
          
          daspilker Daniel Spilker added a comment - The retry in not working for me either. I'll try to give a better analysis. I'm running Jenkins 1.554.2, git-plugin 2.2.1, git-client-plugin 1.9.1. A failed checkout causes a GitException as shown below. GitException is a RuntimeException. That exception is propagated all the way up to the retry handling in AbstractBuildExecution.defaultCheckout. But there only AbortException, InterruptedIOException and IOException trigger a retry. So the problem is that either the retry handling should catch any RuntimeException or GitSCM should not throw a RuntimeException in this case. hudson.plugins.git.GitException: Failed to fetch from https: //github.com/foo/bar.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580) at hudson.model.Run.execute(Run.java:1684) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231)

          We (Ericsson) would like to support that issue. We have a quite large Gerrit infrastructure that require replication and fetch from mirrors only. In that context it happen that the super project is replicated prior to the submodules, thus causing fail checkout. The SCM checkout retry option is not working in this specific case.

          chris31421 Christian Lague added a comment - We (Ericsson) would like to support that issue. We have a quite large Gerrit infrastructure that require replication and fetch from mirrors only. In that context it happen that the super project is replicated prior to the submodules, thus causing fail checkout. The SCM checkout retry option is not working in this specific case.

          Daniel (who left a comment in this discussion back in 18/Jun/14) also submitted a pull request: <https://github.com/jenkinsci/git-plugin/pull/237>.

          FWIW, I've merged his branch with the latest master as of this writing (commit dd256b25ad9ad227be4c8d19794c8f1dc5749f92) and it seems to work fine with my setup that very often trips on network errors when updating git repositories.

          luismbo Luís Oliveira added a comment - Daniel (who left a comment in this discussion back in 18/Jun/14) also submitted a pull request: < https://github.com/jenkinsci/git-plugin/pull/237 >. FWIW, I've merged his branch with the latest master as of this writing (commit dd256b25ad9ad227be4c8d19794c8f1dc5749f92) and it seems to work fine with my setup that very often trips on network errors when updating git repositories.

          Code changed in jenkins
          User: Daniel Spilker
          Path:
          src/main/java/hudson/plugins/git/GitSCM.java
          http://jenkins-ci.org/commit/git-plugin/1c1bb616b28ad50764aaaf073dd9eee056a7e5f6
          Log:
          [fix JENKINS-20531], catch GitException during fetch, throw AbortException

          The AbortException is already thrown from another area of the same
          method, so throwing the AbortException is likely no worse than the
          existing AbortException which is thrown (instead of GitException).

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/1c1bb616b28ad50764aaaf073dd9eee056a7e5f6 Log: [fix JENKINS-20531] , catch GitException during fetch, throw AbortException The AbortException is already thrown from another area of the same method, so throwing the AbortException is likely no worse than the existing AbortException which is thrown (instead of GitException).

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/hudson/plugins/git/GitSCM.java
          http://jenkins-ci.org/commit/git-plugin/72205a77c3b0bcc6feae32a7e149ed5bfae38b73
          Log:
          Add AbortException comment for retry

          Attempt to avoid losing the fix for JENKINS-20531 in future
          changes. No easy way to create a test which asserts an error is thrown
          when fetch() throws an exception, so insert the comment as a weaker
          defense against regression.

          Compare: https://github.com/jenkinsci/git-plugin/compare/4a4433295a71...72205a77c3b0

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/72205a77c3b0bcc6feae32a7e149ed5bfae38b73 Log: Add AbortException comment for retry Attempt to avoid losing the fix for JENKINS-20531 in future changes. No easy way to create a test which asserts an error is thrown when fetch() throws an exception, so insert the comment as a weaker defense against regression. Compare: https://github.com/jenkinsci/git-plugin/compare/4a4433295a71...72205a77c3b0

          Code changed in jenkins
          User: Daniel Spilker
          Path:
          src/main/java/hudson/plugins/git/GitSCM.java
          http://jenkins-ci.org/commit/git-plugin/b6174a1ec3697358435eb691a76f0304b7a9f0ea
          Log:
          [fix JENKINS-20531], catch GitException during fetch, throw AbortException

          The AbortException is already thrown from another area of the same
          method, so throwing the AbortException is likely no worse than the
          existing AbortException which is thrown (instead of GitException).

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Spilker Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/b6174a1ec3697358435eb691a76f0304b7a9f0ea Log: [fix JENKINS-20531] , catch GitException during fetch, throw AbortException The AbortException is already thrown from another area of the same method, so throwing the AbortException is likely no worse than the existing AbortException which is thrown (instead of GitException).

          Code changed in jenkins
          User: Mark Waite
          Path:
          src/main/java/hudson/plugins/git/GitSCM.java
          http://jenkins-ci.org/commit/git-plugin/c3a4092ed6de82224575fea0cf12a57136643352
          Log:
          Add AbortException comment for retry

          Attempt to avoid losing the fix for JENKINS-20531 in future
          changes. No easy way to create a test which asserts an error is thrown
          when fetch() throws an exception, so insert the comment as a weaker
          defense against regression.

          Compare: https://github.com/jenkinsci/git-plugin/compare/d4dda4b56647...c3a4092ed6de

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: src/main/java/hudson/plugins/git/GitSCM.java http://jenkins-ci.org/commit/git-plugin/c3a4092ed6de82224575fea0cf12a57136643352 Log: Add AbortException comment for retry Attempt to avoid losing the fix for JENKINS-20531 in future changes. No easy way to create a test which asserts an error is thrown when fetch() throws an exception, so insert the comment as a weaker defense against regression. Compare: https://github.com/jenkinsci/git-plugin/compare/d4dda4b56647...c3a4092ed6de
          jglick Jesse Glick added a comment -

          Is there a reason this issue is still marked Open?

          The fix seems a bit off. Either defaultCheckout should catch general Exception, and/or GitException should be made to extend IOException rather than RuntimeException.

          jglick Jesse Glick added a comment - Is there a reason this issue is still marked Open? The fix seems a bit off. Either defaultCheckout should catch general Exception , and/or GitException should be made to extend IOException rather than RuntimeException .
          markewaite Mark Waite added a comment -

          It is only marked as open because I haven't been through the fixes prepared for the git plugin 2.2.8 release to assure they are all marked as resolved before it is released.

          I am hesitant to change the inheritance of GitException lest that break compatibility for unknown consumers. Is that a compatibility safe change?

          markewaite Mark Waite added a comment - It is only marked as open because I haven't been through the fixes prepared for the git plugin 2.2.8 release to assure they are all marked as resolved before it is released. I am hesitant to change the inheritance of GitException lest that break compatibility for unknown consumers. Is that a compatibility safe change?
          jglick Jesse Glick added a comment -

          It would not be source-compatible. Whether this is a problem or not is a matter of judgement; arguably a caller of Git commands ought to be aware of the possibility that a command could fail and have thought about how that should be handled.

          jglick Jesse Glick added a comment - It would not be source-compatible. Whether this is a problem or not is a matter of judgement; arguably a caller of Git commands ought to be aware of the possibility that a command could fail and have thought about how that should be handled.
          markewaite Mark Waite added a comment -

          Would this pull request improve the code (better to do it before I release 2.2.8, rather than after)?

          markewaite Mark Waite added a comment - Would this pull request improve the code (better to do it before I release 2.2.8, rather than after)?
          markewaite Mark Waite added a comment -

          Fix released in git plugin 2.3 10 Nov 2014

          markewaite Mark Waite added a comment - Fix released in git plugin 2.3 10 Nov 2014
          markewaite Mark Waite added a comment -

          Fix also included in git plugin 2.2.8 released 12 Nov 2014

          markewaite Mark Waite added a comment - Fix also included in git plugin 2.2.8 released 12 Nov 2014
          huberts Hubert S added a comment -

          Where do you set the Retry Count in jenkins for git to retry?

          huberts Hubert S added a comment - Where do you set the Retry Count in jenkins for git to retry?

          I don't think there's a way to customize it on a per job basis, but you can set a global value in "Manage Jenkins" > "Configure System" > "SCM checkout retry count". HTH.

          luismbo Luís Oliveira added a comment - I don't think there's a way to customize it on a per job basis, but you can set a global value in "Manage Jenkins" > "Configure System" > "SCM checkout retry count". HTH.

          People

            ndeloof Nicolas De Loof
            sapone Sergey Mylnikov
            Votes:
            7 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: