Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36956

Pipeline hangs on JUnitResultArchiver step with test publishers

      This occurs both in parallel and in sequential runs.

      stage 'Unit tests'
      node {
          sh './tools/test/run_unit_tests.sh'
          step([$class: 'JUnitResultArchiver', allowEmptyResults: true, testDataPublishers: [[$class: 'StabilityTestDataPublisher']], testResults: '**/reports/junit-unit.xml'])
      }
      

      when executing following stage/step it hangs with last messages:

      [Pipeline] stage (Unit tests)
      Entering stage Unit tests
      Proceeding
      [Pipeline] node
      Running on мастер in /var/lib/jenkins/home/jobs/project/branches/master/workspace
      [Pipeline] {
      [Pipeline] sh
      [workspace] Running shell script
      + ./tools/test/run_unit_tests.sh
      Starting backend unit tests
      ============================= test session starts ==============================
      platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
      rootdir: /var/lib/jenkins/home/jobs/project/branches/master/workspace, inifile: 
      plugins: asyncio-0.4.1, cov-2.3.0, mock-1.1
      collected 2 items
      
      src/tests/unit/magic.py ..
      
       generated xml file: /var/lib/jenkins/home/jobs/project/branches/master/workspace/reports/junit-unit.xml 
      
      ----------- coverage: platform linux, python 3.5.2-final-0 -----------
      Coverage XML written to file reports/coverage-unit.xml
      
      =========================== 2 passed in 0.31 seconds ===========================
      Finished backend unit tests
      [Pipeline] step
      Recording test results
      

      after that spinner spins forever

          [JENKINS-36956] Pipeline hangs on JUnitResultArchiver step with test publishers

          Daniel Sobral added a comment -

          It stops hanging if you remove test stability.

          Daniel Sobral added a comment - It stops hanging if you remove test stability.

          Oleg Korsak added a comment -

          Yeah, but still this is a code from jenkins sample snippets. And actually removing functionality is not a FIX

          Oleg Korsak added a comment - Yeah, but still this is a code from jenkins sample snippets. And actually removing functionality is not a FIX

          The following hangs for me:

          step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml, **/target/failsafe-reports/TEST-*.xml'])
          

          Jenkins ver 2.10
          Pipeline / Pipeline: Build Step ver 2.2

          Michael Scharp added a comment - The following hangs for me: step([$class: 'JUnitResultArchiver' , testResults: '**/target/surefire-reports/TEST-*.xml, **/target/failsafe-reports/TEST-*.xml' ]) Jenkins ver 2.10 Pipeline / Pipeline: Build Step ver 2.2

          Edward Easton added a comment - - edited

          I can reproduce this two ways - one using the old syntax and one with the new:

          New Syntax as per the snippet generator - this one hangs:

          stage('stage') {
              node {
                  sh "cp /tmp/junit*.xml ."    //  I have a junit file saved from a previous run here
                  junit testDataPublishers: [[$class: 'ClaimTestDataPublisher']], testResults: '**/junit*.xml'
              }
          }
          

          Old syntax - this one works OK

          stage('stage') {
              node {
                  sh "cp /tmp/junit*.xml ."    //  I have a junit file saved from a previous run here
                  step([$class: 'JUnitResultArchiver', testResults: '**/junit*.xml'])
              }
          }
          

          Here is the thread dump of the hung execution:

          Thread #6
          	at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#15])
          	at DSL.node(running on cn13)
          	at WorkflowScript.run(WorkflowScript:2)
          	at DSL.stage(Native Method)
          	at WorkflowScript.run(WorkflowScript:1)
          

          Versions:
          Jenkins - 2.7.4
          Pipeline - 2.4
          Pipeline Build Step - 2.2

          Edward Easton added a comment - - edited I can reproduce this two ways - one using the old syntax and one with the new: New Syntax as per the snippet generator - this one hangs: stage( 'stage' ) { node { sh "cp /tmp/junit*.xml ." // I have a junit file saved from a previous run here junit testDataPublishers: [[$class: 'ClaimTestDataPublisher' ]], testResults: '**/junit*.xml' } } Old syntax - this one works OK stage( 'stage' ) { node { sh "cp /tmp/junit*.xml ." // I have a junit file saved from a previous run here step([$class: 'JUnitResultArchiver' , testResults: '**/junit*.xml' ]) } } Here is the thread dump of the hung execution: Thread #6 at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#15]) at DSL.node(running on cn13) at WorkflowScript.run(WorkflowScript:2) at DSL.stage(Native Method) at WorkflowScript.run(WorkflowScript:1) Versions: Jenkins - 2.7.4 Pipeline - 2.4 Pipeline Build Step - 2.2

          I'm experiencing the same behavior as Edward here above.

             stage('Results') {
                junit '**/target/surefire-reports/TEST-*.xml'  // --> WORKS FINE
          
                junit testDataPublishers: [[$class: 'ClaimTestDataPublisher']], testResults: '**/target/surefire-reports/TEST-*.xml'  // --> HANGS
             }
          

          Thread dump:

          Thread #8
          	at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#13])
          	at WorkflowScript.run(WorkflowScript:20)
          	at DSL.stage(Native Method)
          	at WorkflowScript.run(WorkflowScript:19)
          	at DSL.node(running on )
          	at WorkflowScript.run(WorkflowScript:1)
          

          Richard Kettelerij added a comment - I'm experiencing the same behavior as Edward here above. stage( 'Results' ) { junit '**/target/surefire-reports/TEST-*.xml' // --> WORKS FINE junit testDataPublishers: [[$class: 'ClaimTestDataPublisher' ]], testResults: '**/target/surefire-reports/TEST-*.xml' // --> HANGS } Thread dump: Thread #8 at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#13]) at WorkflowScript.run(WorkflowScript:20) at DSL.stage(Native Method) at WorkflowScript.run(WorkflowScript:19) at DSL.node(running on ) at WorkflowScript.run(WorkflowScript:1)

          eeaston I've traced down the issue we (apparently) both encountered and it was caused by the claim-plugin itself. I've contributed a fix to the claim-plugin to resolve this issue (https://github.com/jenkinsci/claim-plugin/pull/24). This fix is part of claim-plugin v2.9 which is already released. So if you update the claim-plugin your pipeline shouldn't hang on the ClaimTestDataPublisher as it did before.

          Richard Kettelerij added a comment - eeaston I've traced down the issue we (apparently) both encountered and it was caused by the claim-plugin itself. I've contributed a fix to the claim-plugin to resolve this issue ( https://github.com/jenkinsci/claim-plugin/pull/24 ). This fix is part of claim-plugin v2.9 which is already released. So if you update the claim-plugin your pipeline shouldn't hang on the ClaimTestDataPublisher as it did before.

          Sean Flanigan added a comment -

          The claim-plugin issue reported by eeaston and rickette seems to be JENKINS-39537.

          mscharp, you don't seem to be using Test Stability Plugin in that pipeline. I recommend you submit a separate bug report for junit-plugin (including logs) if you're still having that problem.

          kamikaze dcsobral Have you got any more details on the JUnitResultArchiver hang with StabilityTestDataPublisher? Did you have a lot of build history? Could it be JENKINS-33168?

          Sean Flanigan added a comment - The claim-plugin issue reported by eeaston and rickette seems to be JENKINS-39537 . mscharp , you don't seem to be using Test Stability Plugin in that pipeline. I recommend you submit a separate bug report for junit-plugin (including logs) if you're still having that problem. kamikaze dcsobral Have you got any more details on the JUnitResultArchiver hang with StabilityTestDataPublisher? Did you have a lot of build history? Could it be JENKINS-33168 ?

          John Stephenson added a comment - - edited

          Adding this comment incase it helps shed some light on a possible solution or helps others wondering if this is still an issue.

          I also recently started getting this issue in an older version of Jenkins, upgraded (to Jenkins v2.73.2) and issue still present. I'm running 2 parallel builds but they run one after after another ATM as I've only one executor node. Both builds publish using the newer junit syntax. The second one consistently fails ATM.

          Changing this to the old syntax as suggested by Richard K makes it work as before.... for a bit ,but then it's not working now. Race condition?

           

          John Stephenson added a comment - - edited Adding this comment incase it helps shed some light on a possible solution or helps others wondering if this is still an issue. I also recently started getting this issue in an older version of Jenkins, upgraded (to Jenkins v2.73.2) and issue still present. I'm running 2 parallel builds but they run one after after another ATM as I've only one executor node. Both builds publish using the newer junit syntax. The second one consistently fails ATM. Changing this to the old syntax as suggested by Richard K makes it work as before.... for a bit ,but then it's not working now. Race condition?  

          Nikolas Falco added a comment - - edited

          We have the same issue where in a pipeline we run in parallel two maven build that perform integration tests on different application server. We would like collect the tests distinguishing (because are the same) them by branch. When maven build in all branches ends the pipeline hang during the report generation I suppose.

           

          We had try both:

          junit testDataPublishers: [[$class: 'JUnitFlakyTestDataPublisher']], testResults: '**/target/surefire-reports/*.xml'

          and

          junit '**/target/surefire-reports/*.xml'

          Nikolas Falco added a comment - - edited We have the same issue where in a pipeline we run in parallel two maven build that perform integration tests on different application server. We would like collect the tests distinguishing (because are the same) them by branch. When maven build in all branches ends the pipeline hang during the report generation I suppose.   We had try both: junit testDataPublishers: [[$class: 'JUnitFlakyTestDataPublisher' ]], testResults: '**/target/surefire-reports/*.xml' and junit '**/target/surefire-reports/*.xml'

            Unassigned Unassigned
            kamikaze Oleg Korsak
            Votes:
            9 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated: