Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39194

Honor SCM checkout retry count from CpsScmFlowDefinition

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • workflow-cps-plugin
    • None
    • Pipeline Groovy 2.44

      Hi Team,

      We have set the value of SCM checkout retry count to 5. This is working well for free style jobs. The same is not working while loading pipeline using "Pipeline script from scm" option.

      This could be a bug. Can you please check into this.

      Thanks!

        1. Pipeline_Configuration.PNG
          Pipeline_Configuration.PNG
          38 kB
        2. Pipeline_Output.PNG
          Pipeline_Output.PNG
          117 kB
        3. FreeStyle_Output.PNG
          FreeStyle_Output.PNG
          159 kB
        4. FreeStyle_Configuration.PNG
          FreeStyle_Configuration.PNG
          48 kB
        5. SCMRetryCount.PNG
          SCMRetryCount.PNG
          13 kB

          [JENKINS-39194] Honor SCM checkout retry count from CpsScmFlowDefinition

          Mark Waite added a comment -

          anudeeplalam can you provide more details, including step by step instructions and specific examples so that others can duplicate the problem you are reporting?

          For example, please provide the pipeline script you're using which is not honoring the retry count, a description of how you detect the failure when the retry count is not honored, a detailed description of how you configured the working freestyle job (or a copy of the job configuration file), and the types of failures which cause the job to retry.

          Mark Waite added a comment - anudeeplalam can you provide more details, including step by step instructions and specific examples so that others can duplicate the problem you are reporting? For example, please provide the pipeline script you're using which is not honoring the retry count, a description of how you detect the failure when the retry count is not honored, a detailed description of how you configured the working freestyle job (or a copy of the job configuration file), and the types of failures which cause the job to retry.

          Anudeep Lalam added a comment -

          markewaite - Thanks for your comment. Here is the scenario.

          In Manage Jenkins -> Configure System, I have put the SCM checkout retry count to 5.

          Now to test the retry, I have created a free style job to clone a Git repository https://github.com/anudeeplalam/JenkinsTest1.git , which does not exist.

          When I build this job, As you can see in the below picture, The retry is happening.(The retry happened 5 times. To keep short I have captured single retry.)

          Now, When I try to use this in a Pipeline Job for loading Jenkinsfile using "Pipeline script from scm", it's not working

          As you can see in the below picture, The pipeline job exited immediately after the first attempt. Ideally it has to retry "5 Times" just like the Free style Job.

          Anudeep Lalam added a comment - markewaite - Thanks for your comment. Here is the scenario. In Manage Jenkins -> Configure System, I have put the SCM checkout retry count to 5. Now to test the retry, I have created a free style job to clone a Git repository https://github.com/anudeeplalam/JenkinsTest1.git , which does not exist. When I build this job, As you can see in the below picture, The retry is happening.(The retry happened 5 times. To keep short I have captured single retry.) Now, When I try to use this in a Pipeline Job for loading Jenkinsfile using "Pipeline script from scm", it's not working As you can see in the below picture, The pipeline job exited immediately after the first attempt. Ideally it has to retry "5 Times" just like the Free style Job.

          Anudeep Lalam added a comment -

          markewaite - Sorry for a very big message. But I thought it would be better to show all the steps I followed so that it will be easy for you to understand what went wrong.

          So, now coming to the point. Am I missing something here. Ideally my expectation is SCM checkout retry count should work for both Freestyle and Pipeline Jobs.

          This is really a blocker for us. We would really appreciate if someone look into this.

          Thanks!

          Anudeep Lalam added a comment - markewaite - Sorry for a very big message. But I thought it would be better to show all the steps I followed so that it will be easy for you to understand what went wrong. So, now coming to the point. Am I missing something here. Ideally my expectation is SCM checkout retry count should work for both Freestyle and Pipeline Jobs. This is really a blocker for us. We would really appreciate if someone look into this. Thanks!

          Mark Waite added a comment -

          Thanks for the description. That's exactly what is needed.

          I don't expect to investigate it further for at least several months due to other, more common use cases which need investigation and fixes. I wanted enough of a description in the bug report that others might be able to investigate if they encounter the same problem.

          I don't know if the global SCM retry setting that you are using should be honored in pipeline jobs or not. I'm not sure which layer of the Jenkins code implements that retry mechanism. I suspect it is in some component outside the git plugin, since it mentions "SCM checkout retry count" rather than "Git retry count".

          Mark Waite added a comment - Thanks for the description. That's exactly what is needed. I don't expect to investigate it further for at least several months due to other, more common use cases which need investigation and fixes. I wanted enough of a description in the bug report that others might be able to investigate if they encounter the same problem. I don't know if the global SCM retry setting that you are using should be honored in pipeline jobs or not. I'm not sure which layer of the Jenkins code implements that retry mechanism. I suspect it is in some component outside the git plugin, since it mentions "SCM checkout retry count" rather than "Git retry count".

          Jesse Glick added a comment -

          CpsScmFlowDefinition is merely a convenience. You can always write a CpsFlowDefinition (inline script) that does an explicit checkout wrapped in retry.

          Jesse Glick added a comment - CpsScmFlowDefinition is merely a convenience. You can always write a CpsFlowDefinition (inline script) that does an explicit checkout wrapped in retry .

          jglick I'll take a look at this.  It's a blocker for us right now.

          Matthew Mitchell added a comment - jglick  I'll take a look at this.  It's a blocker for us right now.

          My take is that the retry count should be at least honored for the pipeline scm checkout, though not for checkout steps within the pipeline script.

          Matthew Mitchell added a comment - My take is that the retry count should be at least honored for the pipeline scm checkout, though not for checkout steps within the pipeline script.

          Jesse Glick added a comment -

          Yeah I would say CpsScmFlowDefinition could pay attention to GlobalSCMRetryCountConfiguration. Within a build, you can just use retry as needed.

          Jesse Glick added a comment - Yeah I would say CpsScmFlowDefinition could pay attention to GlobalSCMRetryCountConfiguration . Within a build, you can just use retry as needed.

          jglick This should be straightforward, though I was also contemplating adding the retry count override option (like with freestyle jobs) into the CpsScmFlowDefinition UI.  Thoughts?  It's probably not necessary but it could be useful to someone.

          Matthew Mitchell added a comment - jglick This should be straightforward, though I was also contemplating adding the retry count override option (like with freestyle jobs) into the CpsScmFlowDefinition UI.  Thoughts?  It's probably not necessary but it could be useful to someone.

          Jesse Glick added a comment -

          As long as it goes into an f:advanced block.

          Jesse Glick added a comment - As long as it goes into an f:advanced block.

          Matthew Mitchell added a comment - Implemented in  https://github.com/jenkinsci/workflow-cps-plugin/pull/147

          The retry feature on Pipelines would be extremely helpful for us.  Our SCM is often unreliable where a checkout will time out:

          stderr: fatal: unable to access 'https://tfs-...': Failed to connect to tfs-... port 443: Timed out

           

          We implement retries around the git fetches completed within the pipeline itself, and typically a single retry will do the trick.  Short of using pull/147 I don't see any way for us to work around the issue of failed git fetches within the (Declarative: Checkout SCM) stage.  We are not interested in installing an unofficial fix.

           

          The fix mentioned on this issue would resolve our current most frequent reason of build failures.  Is there an ETA for resolution?

          Anthony Segretto added a comment - The retry feature on Pipelines would be extremely helpful for us.  Our SCM is often unreliable where a checkout will time out: stderr: fatal: unable to access 'https://tfs-...': Failed to connect to tfs-... port 443: Timed out   We implement retries around the git fetches completed within the pipeline itself, and typically a single retry will do the trick.  Short of using pull/147 I don't see any way for us to work around the issue of failed git fetches within the (Declarative: Checkout SCM) stage.  We are not interested in installing an unofficial fix.   The fix mentioned on this issue would resolve our current most frequent reason of build failures.  Is there an ETA for resolution?

          Minghua Zhao added a comment -

          Is there any update for this issue? I saw the pull request was approved but not merged.

          Minghua Zhao added a comment - Is there any update for this issue? I saw the pull request was approved but not merged.

          Code changed in jenkins
          User: Sam Van Oort
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/d91d14d0323d0944dc5a89fef83c83283c7d13d9
          Log:
          Merge pull request #147 from mmitche/add-retry

          JENKINS-39194 Implement SCM retry count

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/e9b183243542...d91d14d0323d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java src/test/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinitionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/d91d14d0323d0944dc5a89fef83c83283c7d13d9 Log: Merge pull request #147 from mmitche/add-retry JENKINS-39194 Implement SCM retry count Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/e9b183243542...d91d14d0323d

          Felipe Santos added a comment -

          While the global retries configuration is working for the initial checkout (the one that fetches the Jenkinsfile), I noticed that the second checkout (the one on the agent that fetches the whole repository) isn't. I filled a bug report for this at: https://issues.jenkins.io/browse/JENKINS-72429

          Felipe Santos added a comment - While the global retries configuration is working for the initial checkout (the one that fetches the Jenkinsfile), I noticed that the second checkout (the one on the agent that fetches the whole repository) isn't. I filled a bug report for this at: https://issues.jenkins.io/browse/JENKINS-72429

            mmitche Matthew Mitchell
            anudeeplalam Anudeep Lalam
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: