We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

      When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for every git checkout that runs at the same time.

      In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

      One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

      The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.

      lock('checkout') {
          def checkouts = [:]
          def threads = 8
      
          stage('Super Repo') {
              checkout([$class: 'GitSCM',
                      branches: [[name: branch]],
                      doGenerateSubmoduleConfigurations: false,
                      extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                      submoduleCfg: [],
                      userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])
      
              def repos = readFile('.gitslave')
              def reposLines = repos.readLines()
              for (i = 0; i < threads; i++){
                  checkouts["Thread ${i}"] = []
              }
              def idx = 0
              for (line in reposLines) {
                  def repoInfo = line.split(' ')
                  def repoUrl = repoInfo[0]
                  def repoPath = repoInfo[1]
                  def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
                  def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
                  echo("Thread ${idx%threads}")
                  checkouts["Thread ${idx%threads}"] << [path: curatedRepoPath, url: "${gitBaseUrl}/${curatedRepoUrl}"]
                  idx++
              }
          }
          stage('GitSlave Repos') {
              def doCheckouts = [:]
              for (i = 0; i < threads; i++){
                  def j = i
                  doCheckouts["Thread ${j}"] = {
                      for (co in checkouts["Thread ${j}"]) {
                          retry(3) {
                              checkout([$class: 'GitSCM',
                                      branches: [[name: branch]],
                                      doGenerateSubmoduleConfigurations: false,
                                      extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: co.path], [$class: 'CleanCheckout'], [$class: 'ScmName', name: co.path]],
                                      submoduleCfg: [],
                                      userRemoteConfigs: [[url: co.url]]])
                          }
                      }
                  }
              }
              parallel doCheckouts
          }
      

          [JENKINS-39968] git checkouts are not pipeline-parallel safe

          Julien Pivotto created issue -
          Julien Pivotto made changes -
          Description Original: We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

          When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

          In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

          One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.
          New: We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

          When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

          In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

          One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

          The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.
          Julien Pivotto made changes -
          Description Original: We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

          When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

          In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

          One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

          The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.
          New: We have a specific workflow that enables us to discover that bug. However, it could and will probably happen to other people soon.

          When running scm checkouts in parallels in a Jenkins file, the changes discovered are discovered for *every* git checkout that runs at the same time.

          In our example, we run checkouts with 8 threads in parallel (background: https://roidelapluie.be/blog/2016/11/18/gitslave-jenkins/ ).

          One change is displayed in 8 different repositories, the 8 repositories that are checked out at the same time.

          The next checkouts do not contains the changes, so I guess it is kind of a global variable that is reset after the checkout.



          {code:java}
          lock('checkout') {
              def checkouts = [:]
              def threads = 8

              stage('Super Repo') {
                  checkout([$class: 'GitSCM',
                          branches: [[name: branch]],
                          doGenerateSubmoduleConfigurations: false,
                          extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                          submoduleCfg: [],
                          userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])

                  def repos = readFile('.gitslave')
                  def reposLines = repos.readLines()
                  for (i = 0; i < threads; i++){
                      checkouts["Thread ${i}"] = []
                  }
                  def idx = 0
                  for (line in reposLines) {
                      def repoInfo = line.split(' ')
                      def repoUrl = repoInfo[0]
                      def repoPath = repoInfo[1]
                      def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
                      def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
                      echo("Thread ${idx%threads}")
                      checkouts["Thread ${idx%threads}"] << [path: curatedRepoPath, url: "${gitBaseUrl}/${curatedRepoUrl}"]
                      idx++
                  }
              }
              stage('GitSlave Repos') {
                  def doCheckouts = [:]
                  for (i = 0; i < threads; i++){
                      def j = i
                      doCheckouts["Thread ${j}"] = {
                          for (co in checkouts["Thread ${j}"]) {
                              retry(3) {
                                  checkout([$class: 'GitSCM',
                                          branches: [[name: branch]],
                                          doGenerateSubmoduleConfigurations: false,
                                          extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: co.path], [$class: 'CleanCheckout'], [$class: 'ScmName', name: co.path]],
                                          submoduleCfg: [],
                                          userRemoteConfigs: [[url: co.url]]])
                              }
                          }
                      }
                  }
                  parallel doCheckouts
              }
          {code}
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]
          Kevin Phillips made changes -
          Link New: This issue is related to JENKINS-54732 [ JENKINS-54732 ]
          Tiago Santos made changes -
          Component/s New: workflow-scm-step-plugin [ 21717 ]
          Devin Nusbaum made changes -
          Link New: This issue is duplicated by JENKINS-55040 [ JENKINS-55040 ]
          Jesse Glick made changes -
          Link New: This issue duplicates JENKINS-34313 [ JENKINS-34313 ]
          Andreas Ringlstetter made changes -
          Link New: This issue is related to JENKINS-58068 [ JENKINS-58068 ]
          Andreas Ringlstetter made changes -
          Link Original: This issue is related to JENKINS-58068 [ JENKINS-58068 ]

            Unassigned Unassigned
            roidelapluie Julien Pivotto
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: