Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60022

Stages running in different agents checkout different versions

      Given a pipeline with multiple stages and some stages running on different agents, if a new commit is push while the pipeline is running, a stage running later in a different agent will implicitly checkout the new commit instead of the commit that originated the pipeline execution.

       

      Steps to reproduce:

      1. Using this repo https://github.com/nicopaez/ejemplo create a pipeline job
      2. Start the pipeline, it will wait for you input after stage 1
      3. Commit+push a change in the repository
      4. Let the pipeline proceed with the execution

      Expected behavior: both stages get the same git commit
      Actual behavior: both stages are getting different git commits

       

      I think the issues is in the implicit checkout operations that is perform on each stage. It seems it is checking out the branch when it should checkout the specific commits that was checkout when the pipeline was started.

       

      We verified this issues on 2 different configurations:

      • A single instance Jenkins setup using docker agents
      • A master-slave Jenkins setup using different server nodes (no docker)

          [JENKINS-60022] Stages running in different agents checkout different versions

          nicopaez did you happen to find the fix/reason for this?

          This behaviour looks very odd.

          Harsh Kulshrestha added a comment - nicopaez did you happen to find the fix/reason for this? This behaviour looks very odd.

          M. B. added a comment -

          nicopaez I had a look into your Jenkinsfile: In your case you should simply add "skipDefaultCheckout()" to your pipeline options.

          Your Docker agents of the stages do in fact just mount your current main agent's workspace ('agent any'). So there is not even the need to do another checkout.

           

          Apart from your specific Jenkinsfile:

          A pipeline is generally configured to look for new commits in a branch. Once a change is detected, the found $GIT_COMMIT is added to the environment. If you issue another "look for changes" and there are some, they will be found. A "checkout scm" does that, implicit or not. It reuses the original pipeline scm definition of what to look for. If you want to checkout a single specific revision again in a later stage, you'll need to do that explicitly, by using the original $GIT_COMMIT var. But that "definition of a checkout" will be a different one than the pipelinie run originally started with.

           

          TLDR: I dont consider this to be a Major Bug at all.

          M. B. added a comment - nicopaez I had a look into your Jenkinsfile: In your case you should simply add "skipDefaultCheckout()" to your pipeline options. Your Docker agents of the stages do in fact just mount your current main agent's workspace ('agent any'). So there is not even the need to do another checkout.   Apart from your specific Jenkinsfile: A pipeline is generally configured to look for new commits in a branch. Once a change is detected, the found $GIT_COMMIT is added to the environment. If you issue another "look for changes" and there are some, they will be found. A "checkout scm" does that, implicit or not. It reuses the original pipeline scm definition of what to look for. If you want to checkout a single specific revision again in a later stage, you'll need to do that explicitly, by using the original $GIT_COMMIT var. But that "definition of a checkout" will be a different one than the pipelinie run originally started with.   TLDR: I dont consider this to be a Major Bug at all.

          Checking out to the latest commit might end up making things complex in projects where a user input is required.

          Jenkins recommends using the parent agent "none" for pipelines with an input step so that the input step doesn't end up blocking an agent. As a result, one needs to specify agents at every other stage, which causes Jenkins to checkout by default.

          And skipDefaultCheckout() might not always work in cases where the pipeline uses multiple agents. If a new agent is allocated to the pipeline in the middle of the build, one will need to pull the changes corresponding to the original commit that triggered the pipeline.

           

          I might be wrong, would love to hear inputs.

          Harsh Kulshrestha added a comment - Checking out to the latest commit might end up making things complex in projects where a user input is required. Jenkins recommends using the parent agent "none" for pipelines with an input step so that the input step doesn't end up blocking an agent. As a result, one needs to specify agents at every other stage, which causes Jenkins to checkout by default. And skipDefaultCheckout() might not always work in cases where the pipeline uses multiple agents. If a new agent is allocated to the pipeline in the middle of the build, one will need to pull the changes corresponding to the original commit that triggered the pipeline.   I might be wrong, would love to hear inputs.

          M. B. added a comment -

          And skipDefaultCheckout() might not always work in cases where the pipeline uses multiple agents. If a new agent is allocated to the pipeline in the middle of the build, one will need to pull the changes corresponding to the original commit that triggered the pipeline.

          In that case, I would assign the original $GIT_COMMIT to a variable and check out that specific revision on that new agent (instead of a branch), together with 'skipDefaultCheckout' as stage option.

          Aside that, for smaller projects and files, you could also utilize stash/unstash to transport files to other agent workspaces.

          M. B. added a comment - And skipDefaultCheckout() might not always work in cases where the pipeline uses multiple agents. If a new agent is allocated to the pipeline in the middle of the build, one will need to pull the changes corresponding to the original commit that triggered the pipeline. In that case, I would assign the original $GIT_COMMIT to a variable and check out that specific revision on that new agent (instead of a branch), together with 'skipDefaultCheckout' as stage option. Aside that, for  smaller projects and files, you could also utilize stash/unstash to transport files to other agent workspaces.

            Unassigned Unassigned
            nicopaez Nicolas Paez
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: