Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45489

checkout(scm) step can return wrong variables when used following another Git checkout

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • scm-api-plugin
    • None
    • git plugin 4.7.0 released 17 Mar 2021

      This looks like the same type of code-path issue described in https://issues.jenkins-ci.org/browse/JENKINS-41996 

       

      @Library('someGitLibrary')
      
      node {
        final scmVars = checkout(scm)
        // scmVars may have Git data from either the someGitLibrary or the scmVars
       }

       

      Looks like its caused by https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L1282-L1317 

       

          [JENKINS-45489] checkout(scm) step can return wrong variables when used following another Git checkout

          I can confirm that I'm seeing this with git-plugin 3.5.0 and jenkins 2.60.2 and it seems to be pretty consistent on a particular pipeline job, where I have now replayed the job 10 times with the same output.

          I don't see any indication that the variables are mixed between both repositories, all of them seem to be from the one I import the "library" from.

          In my setup I have a jenkinsfile from a repository which includes a "library" file from the same repository, through the usage of fileloader.withGit. The actual code that I'm building/testing and am checking out with checkout step is a second repository.

          mihail stoykov added a comment - I can confirm that I'm seeing this with git-plugin 3.5.0 and jenkins 2.60.2 and it seems to be pretty consistent on a particular pipeline job, where I have now replayed the job 10 times with the same output. I don't see any indication that the variables are mixed between both repositories, all of them seem to be from the one I import the "library" from. In my setup I have a jenkinsfile from a repository which includes a "library" file from the same repository, through the usage of fileloader.withGit. The actual code that I'm building/testing and am checking out with checkout step is a second repository.

          Mark Wright added a comment - - edited

          The issue is repeatable without involving shared-libraries. Simply by having an SCM pipeline script which checks out a different revision to that used to load the pipeline sciprt. The checkout step returns the GIT_COMMIT of the original checkout and not the new checkout.

          I've also seen this manifested in my multibranch pipeline jobs, but it is best demonstrated by a simple parameterised pipeline.

          Create a Pipeline job configured with "refs/heads/master" as the branch specifier. Give the job a single "TAG" parameter, providing a tag to be checked out by the pipeline. Use this script:

          pipeline
          {
              agent any
              environment
              {
                  git_tool = tool(name: 'Default', type: 'git')
              }
              options
              {
                  skipDefaultCheckout()
              }
              stages
              {
                  stage('Checkout')
                  {
                      steps
                      {
                          script
                          {
                              def buildTag = env.TAG
                              echo "Building against tag '${buildTag}'"
          
                              def newScm = [$class: 'GitSCM', userRemoteConfigs: scm.userRemoteConfigs, branches: [[name: "refs/tags/${buildTag}"]]]
          
                              def checkoutDetails = checkout scm: newScm, poll: false, changelog: false
          
                              echo "checkout scm returned SHA = ${checkoutDetails.GIT_COMMIT}"
                              bat """git status"""
                          }
                      }
                  }
              }
          }
          

          When run with a tag at the tip of the specified branch (master) we see the returned SHA is as expected:

          Building against tag 'tag3'
          [Pipeline] checkout
           > C:\Program Files\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=120
          Fetching changes from the remote Git repository
           > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@github.aus.thenational.com:P640806/jenkins_testing.git # timeout=120
          Fetching upstream changes from git@github.aus.thenational.com:P640806/jenkins_testing.git
           > C:\Program Files\Git\cmd\git.exe --version # timeout=120
          using GIT_SSH to set credentials 
           > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git@github.aus.thenational.com:P640806/jenkins_testing.git +refs/heads/*:refs/remotes/origin/*
           > C:\Program Files\Git\cmd\git.exe rev-parse "refs/tags/tag3^{commit}" # timeout=120
           > C:\Program Files\Git\cmd\git.exe rev-parse "refs/remotes/origin/refs/tags/tag3^{commit}" # timeout=120
          Checking out Revision e03409cea735e7353e421551c6bd5963f436c38a (refs/tags/tag3)
           > C:\Program Files\Git\cmd\git.exe config core.sparsecheckout # timeout=120
           > C:\Program Files\Git\cmd\git.exe checkout -f e03409cea735e7353e421551c6bd5963f436c38a
          Commit message: "Removed env.GIT_COMMIT usage"
          [Pipeline] echo
          checkout scm returned SHA = e03409cea735e7353e421551c6bd5963f436c38a
          [Pipeline] bat
          [test_scm_issue] Running batch script
          
          d:\jenkins\ws\test_scm_issue>git status 
          HEAD detached at e03409c
          nothing to commit, working tree clean
          

          If an additional commit is added to master (so the tag is no longer at the tip), we see the correct revision (e03409c) is checked out, however the revision state returned by the checkout step is the new tip of master (5e70b9e903582cd1055e60b819c6f2f09f01aee7):

          Building against tag 'tag3'
          [Pipeline] checkout
           > C:\Program Files\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=120
          Fetching changes from the remote Git repository
           > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@github.aus.thenational.com:P640806/jenkins_testing.git # timeout=120
          Fetching upstream changes from git@github.aus.thenational.com:P640806/jenkins_testing.git
           > C:\Program Files\Git\cmd\git.exe --version # timeout=120
          using GIT_SSH to set credentials 
           > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git@github.aus.thenational.com:P640806/jenkins_testing.git +refs/heads/*:refs/remotes/origin/*
           > C:\Program Files\Git\cmd\git.exe rev-parse "refs/tags/tag3^{commit}" # timeout=120
           > C:\Program Files\Git\cmd\git.exe rev-parse "refs/remotes/origin/refs/tags/tag3^{commit}" # timeout=120
          Checking out Revision e03409cea735e7353e421551c6bd5963f436c38a (refs/tags/tag3)
           > C:\Program Files\Git\cmd\git.exe config core.sparsecheckout # timeout=120
           > C:\Program Files\Git\cmd\git.exe checkout -f e03409cea735e7353e421551c6bd5963f436c38a
          Commit message: "Removed env.GIT_COMMIT usage"
          [Pipeline] echo
          checkout scm returned SHA = 5e70b9e903582cd1055e60b819c6f2f09f01aee7
          [Pipeline] bat
          [test_scm_issue] Running batch script
          
          d:\jenkins\ws\test_scm_issue>git status 
          HEAD detached at e03409c
          nothing to commit, working tree clean
          

          Mark Wright added a comment - - edited The issue is repeatable without involving shared-libraries. Simply by having an SCM pipeline script which checks out a different revision to that used to load the pipeline sciprt. The checkout step returns the GIT_COMMIT of the original checkout and not the new checkout. I've also seen this manifested in my multibranch pipeline jobs, but it is best demonstrated by a simple parameterised pipeline. Create a Pipeline job configured with " refs/heads/master " as the branch specifier. Give the job a single "TAG" parameter, providing a tag to be checked out by the pipeline. Use this script: pipeline { agent any environment { git_tool = tool(name: 'Default' , type: 'git' ) } options { skipDefaultCheckout() } stages { stage( 'Checkout' ) { steps { script { def buildTag = env.TAG echo "Building against tag '${buildTag}' " def newScm = [$class: 'GitSCM' , userRemoteConfigs: scm.userRemoteConfigs, branches: [[name: "refs/tags/${buildTag}" ]]] def checkoutDetails = checkout scm: newScm, poll: false , changelog: false echo "checkout scm returned SHA = ${checkoutDetails.GIT_COMMIT}" bat """git status" "" } } } } } When run with a tag at the tip of the specified branch (master) we see the returned SHA is as expected: Building against tag 'tag3' [Pipeline] checkout > C:\Program Files\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=120 Fetching changes from the remote Git repository > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@github.aus.thenational.com:P640806/jenkins_testing.git # timeout=120 Fetching upstream changes from git@github.aus.thenational.com:P640806/jenkins_testing.git > C:\Program Files\Git\cmd\git.exe --version # timeout=120 using GIT_SSH to set credentials > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git@github.aus.thenational.com:P640806/jenkins_testing.git +refs/heads/*:refs/remotes/origin/* > C:\Program Files\Git\cmd\git.exe rev-parse "refs/tags/tag3^{commit}" # timeout=120 > C:\Program Files\Git\cmd\git.exe rev-parse "refs/remotes/origin/refs/tags/tag3^{commit}" # timeout=120 Checking out Revision e03409cea735e7353e421551c6bd5963f436c38a (refs/tags/tag3) > C:\Program Files\Git\cmd\git.exe config core.sparsecheckout # timeout=120 > C:\Program Files\Git\cmd\git.exe checkout -f e03409cea735e7353e421551c6bd5963f436c38a Commit message: "Removed env.GIT_COMMIT usage" [Pipeline] echo checkout scm returned SHA = e03409cea735e7353e421551c6bd5963f436c38a [Pipeline] bat [test_scm_issue] Running batch script d:\jenkins\ws\test_scm_issue>git status HEAD detached at e03409c nothing to commit, working tree clean If an additional commit is added to master (so the tag is no longer at the tip), we see the correct revision (e03409c) is checked out, however the revision state returned by the checkout step is the new tip of master (5e70b9e903582cd1055e60b819c6f2f09f01aee7): Building against tag 'tag3' [Pipeline] checkout > C:\Program Files\Git\cmd\git.exe rev-parse --is-inside-work-tree # timeout=120 Fetching changes from the remote Git repository > C:\Program Files\Git\cmd\git.exe config remote.origin.url git@github.aus.thenational.com:P640806/jenkins_testing.git # timeout=120 Fetching upstream changes from git@github.aus.thenational.com:P640806/jenkins_testing.git > C:\Program Files\Git\cmd\git.exe --version # timeout=120 using GIT_SSH to set credentials > C:\Program Files\Git\cmd\git.exe fetch --tags --progress git@github.aus.thenational.com:P640806/jenkins_testing.git +refs/heads/*:refs/remotes/origin/* > C:\Program Files\Git\cmd\git.exe rev-parse "refs/tags/tag3^{commit}" # timeout=120 > C:\Program Files\Git\cmd\git.exe rev-parse "refs/remotes/origin/refs/tags/tag3^{commit}" # timeout=120 Checking out Revision e03409cea735e7353e421551c6bd5963f436c38a (refs/tags/tag3) > C:\Program Files\Git\cmd\git.exe config core.sparsecheckout # timeout=120 > C:\Program Files\Git\cmd\git.exe checkout -f e03409cea735e7353e421551c6bd5963f436c38a Commit message: "Removed env.GIT_COMMIT usage" [Pipeline] echo checkout scm returned SHA = 5e70b9e903582cd1055e60b819c6f2f09f01aee7 [Pipeline] bat [test_scm_issue] Running batch script d:\jenkins\ws\test_scm_issue>git status HEAD detached at e03409c nothing to commit, working tree clean

          Richard Bowater added a comment - - edited

          I've just run up against this too.

          Fetching changes from the remote Git repository
          Fetching without tags
          Checking out Revision 5cc3005a298d50fa0c2bc919293fcf3b3ed936de (print-env)
          Commit message: "Print more env"
          
          GIT_COMMIT=eb65cd4f46c828362429cbbd2d904f114bb4b801
          GIT_BRANCH=master
          

          Doesn't seem to be an obvious way to work around it? My particular use case is:

          • a global shared library configured to checkout from git
          • a multibranch pipeline configured to load Jenkinsfiles from the shared libraries above

          We're using pipelines to build and test shared libraries before deployment, but branch builds are being unpredictable due to the environment variables being incorrect.

          Richard Bowater added a comment - - edited I've just run up against this too. Fetching changes from the remote Git repository Fetching without tags Checking out Revision 5cc3005a298d50fa0c2bc919293fcf3b3ed936de (print-env) Commit message: "Print more env" GIT_COMMIT=eb65cd4f46c828362429cbbd2d904f114bb4b801 GIT_BRANCH=master Doesn't seem to be an obvious way to work around it? My particular use case is: a global shared library configured to checkout from git a multibranch pipeline configured to load Jenkinsfiles from the shared libraries above We're using pipelines to build and test shared libraries before deployment, but branch builds are being unpredictable due to the environment variables being incorrect.

          Mark Waite added a comment - - edited

          rjbwtr the most direct work around is to compute the SHA1 of the working directory with a call to a shell script, batch script, or powershell script. Refer to the example in a stackoverflow article.

          The technique is also in the pipeline script utilities that I use.

          Mark Waite added a comment - - edited rjbwtr the most direct work around is to compute the SHA1 of the working directory with a call to a shell script, batch script, or powershell script. Refer to the example in a stackoverflow article . The technique is also in the pipeline script utilities that I use.

          Mark Wright added a comment -

          rjbwtr, on (Windows) build-server, we use the following in our scripts as markewaite suggests:

          def sha = bat(returnStdout: true, script: '@git rev-parse HEAD').trim() 

          Mark Wright added a comment - rjbwtr , on (Windows) build-server, we use the following in our scripts as markewaite suggests: def sha = bat(returnStdout: true , script: '@git rev-parse HEAD' ).trim()

          Bram Mertens added a comment -

          This is not limited to git.

          We see the same or a very similar issue using an SVN repo (which includes svn:externals) and shared libraries stored in SVN.
          The values returned by def checkoutResults = checkout(scm ...) contain SVN revision numbers that are not the SVN revision checked out.
          Since we use this information in the generated Manifest files this is a major problem.

          Bram Mertens added a comment - This is not limited to git. We see the same or a very similar issue using an SVN repo (which includes svn:externals) and shared libraries stored in SVN. The values returned by def checkoutResults = checkout(scm ...) contain SVN revision numbers that are not the SVN revision checked out. Since we use this information in the generated Manifest files this is a major problem.

          Fabian Holler added a comment -

          This is a very dangerous bug.
          We are often checking out the Jenkinsfile from branch A and then do in the Jenkinsfile:

          commit = checkout([$class: 'GitSCM', branches: [[name: otherversion]]
          ).GIT_COMMIT

          The returned commit id was the one from the initial SCM checkout for the jenkinsfile, not the one specified as checkout() parameter.

          This caused that we deployed applications in the wrong version.

          Fabian Holler added a comment - This is a very dangerous bug. We are often checking out the Jenkinsfile from branch A and then do in the Jenkinsfile: commit = checkout([$class: 'GitSCM' , branches: [[name: otherversion]] ).GIT_COMMIT The returned commit id was the one from the initial SCM checkout for the jenkinsfile, not the one specified as checkout() parameter. This caused that we deployed applications in the wrong version.

          Also having the issue.
          Any workaround for GIT_PREVIOUS_SUCCESSFUL_COMMIT ?

          Edouard Lavaud added a comment - Also having the issue. Any workaround for GIT_PREVIOUS_SUCCESSFUL_COMMIT ?

          Dee Kryvenko added a comment -

          We had CI pipeline for the shared jenkins lib that uses itself, i.e. new commits being tested and promoted via pipeline using one of the old commits addressed by a tag. Somehow it's being working for years and started to hit us recently. Because of this bug, our pipeline constantly re-promotes what was already there instead of promoting the new code. This sounds like a very critical and dangerous bug as it may let the users promote wrong versions to prod environment and cause an outage.

          Dee Kryvenko added a comment - We had CI pipeline for the shared jenkins lib that uses itself, i.e. new commits being tested and promoted via pipeline using one of the old commits addressed by a tag. Somehow it's being working for years and started to hit us recently. Because of this bug, our pipeline constantly re-promotes what was already there instead of promoting the new code. This sounds like a very critical and dangerous bug as it may let the users promote wrong versions to prod environment and cause an outage.

          Mark Waite added a comment -

          Git plugin 4.7.0 returns the expected value from checkout scm. As far as I can tell, this should be resolved in multibranch pipelines and organization folders thanks to the fix for JENKINS-53346

          Mark Waite added a comment - Git plugin 4.7.0 returns the expected value from checkout scm . As far as I can tell, this should be resolved in multibranch pipelines and organization folders thanks to the fix for JENKINS-53346

            Unassigned Unassigned
            mkobit Mike Kobit
            Votes:
            9 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: