Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39198

External Workspace Manager doesn't work properly with parallel test execution

      The external workspace manager plugin doesn't work very well when using parallel test execution. The main problem is that it creates different workspaces for each test that is executed in parallel.

      Here's a basic example:

      Upstream code:

      node('master') {
          def extWorkspace = exwsAllocate 'dp1'
          exws(extWorkspace) {
              stage "Checkout ${GIT_BRANCH}"
              git ...
      
      
              stage 'Test execution'
              def TEST_CONFIG = "production"
              def tests = ['test_1',
                           'test_2',
      
              ]
              try {
                  def branches = [:]
                  for (int i = 0; i < tests.size(); i++) {
      
                      def s = tests.get(i)
                      def stepName = "running ${s}"
      
                      branches[stepName] = runTestSuite(s, TEST_CONFIG)
                  }
      
                  parallel branches
              } catch (all) {
      
              } finally {
                  stage 'Test results parsing'
                  step([$class: 'JUnitResultArchiver', testResults: 'junit/*.xml'])
              }
          }
      }
      
      def runTestSuite(test_suite, config) {
          return {
              node('master') {
                  build job: 'Run test suite',
                          parameters: [[$class: 'StringParameterValue', name: 'test', value: test_suite],
                                       [$class: 'StringParameterValue', name: 'TEST_CONFIG', value: config],
                                       [$class: 'StringParameterValue', name: 'UPSTREAM_BUILD_NUMBER', value: currentBuild.number.toString()],
                          ]
              }
          }
      }
      

      Downstream code is:

      node('master') {
          def customPath = "tmp/TEST2/${UPSTREAM_BUILD_NUMBER}"
          def extWorkspace = exwsAllocate diskPoolId: 'dp1', path: customPath
      
      
          exws(extWorkspace) {
      
              currentBuild.displayName = "branch: ${GIT_BRANCH}, config: prod, test: ${test}"
      
              stage 'Test execution'
              sh "py.test ... ${test}"
      
          }
      }
      

      Due to parallel execution of downstream job, builds start it in different workspaces, for example:
      /tmp/TEST2/63@2
      /tmp/TEST2/63@3

          [JENKINS-39198] External Workspace Manager doesn't work properly with parallel test execution

          Code changed in jenkins
          User: Alexandru Somai
          Path:
          src/main/java/org/jenkinsci/plugins/ewm/steps/ExwsExecution.java
          http://jenkins-ci.org/commit/external-workspace-manager-plugin/f984366a453ca471fbf9da35a7d7a92cd98a7ff8
          Log:
          JENKINS-39198 Fix workspace allocation - do not lease workspace, since this may not work properly with parallel test execution

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alexandru Somai Path: src/main/java/org/jenkinsci/plugins/ewm/steps/ExwsExecution.java http://jenkins-ci.org/commit/external-workspace-manager-plugin/f984366a453ca471fbf9da35a7d7a92cd98a7ff8 Log: JENKINS-39198 Fix workspace allocation - do not lease workspace, since this may not work properly with parallel test execution

          Code changed in jenkins
          User: Alexandru Somai
          Path:
          src/main/java/org/jenkinsci/plugins/ewm/steps/ExwsExecution.java
          http://jenkins-ci.org/commit/external-workspace-manager-plugin/f0ba616f94167731fe4a478715b515a8de4ce7f4
          Log:
          Merge pull request #48 from alexsomai/fix-parallel-execution

          JENKINS-39198 Fix workspace allocation in parallel test execution

          Compare: https://github.com/jenkinsci/external-workspace-manager-plugin/compare/c4a5d486d326...f0ba616f9416

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alexandru Somai Path: src/main/java/org/jenkinsci/plugins/ewm/steps/ExwsExecution.java http://jenkins-ci.org/commit/external-workspace-manager-plugin/f0ba616f94167731fe4a478715b515a8de4ce7f4 Log: Merge pull request #48 from alexsomai/fix-parallel-execution JENKINS-39198 Fix workspace allocation in parallel test execution Compare: https://github.com/jenkinsci/external-workspace-manager-plugin/compare/c4a5d486d326...f0ba616f9416

            alexsomai Alexandru Somai
            alexsomai Alexandru Somai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: