Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36956

Pipeline hangs on JUnitResultArchiver step with test publishers

    XMLWordPrintable

Details

    Description

      This occurs both in parallel and in sequential runs.

      stage 'Unit tests'
      node {
          sh './tools/test/run_unit_tests.sh'
          step([$class: 'JUnitResultArchiver', allowEmptyResults: true, testDataPublishers: [[$class: 'StabilityTestDataPublisher']], testResults: '**/reports/junit-unit.xml'])
      }
      

      when executing following stage/step it hangs with last messages:

      [Pipeline] stage (Unit tests)
      Entering stage Unit tests
      Proceeding
      [Pipeline] node
      Running on мастер in /var/lib/jenkins/home/jobs/project/branches/master/workspace
      [Pipeline] {
      [Pipeline] sh
      [workspace] Running shell script
      + ./tools/test/run_unit_tests.sh
      Starting backend unit tests
      ============================= test session starts ==============================
      platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
      rootdir: /var/lib/jenkins/home/jobs/project/branches/master/workspace, inifile: 
      plugins: asyncio-0.4.1, cov-2.3.0, mock-1.1
      collected 2 items
      
      src/tests/unit/magic.py ..
      
       generated xml file: /var/lib/jenkins/home/jobs/project/branches/master/workspace/reports/junit-unit.xml 
      
      ----------- coverage: platform linux, python 3.5.2-final-0 -----------
      Coverage XML written to file reports/coverage-unit.xml
      
      =========================== 2 passed in 0.31 seconds ===========================
      Finished backend unit tests
      [Pipeline] step
      Recording test results
      

      after that spinner spins forever

      Attachments

        Issue Links

          Activity

            kamikaze Oleg Korsak created issue -
            kamikaze Oleg Korsak made changes -
            Field Original Value New Value
            Description This occurs both in parallel and in sequential runs.

            {code:groovy}
            stage 'Unit tests'
            node {
                sh './tools/test/run_unit_tests.sh'
                step([$class: 'JUnitResultArchiver', allowEmptyResults: true, testDataPublishers: [[$class: 'StabilityTestDataPublisher']], testResults: '**/reports/junit-unit.xml'])
            }
            {code}
            when executing following stage/step it hangs with last messages:


            {noformat}
            [Pipeline] stage (Unit tests)
            Entering stage Unit tests
            Proceeding
            [Pipeline] node
            Running on мастер in /var/lib/jenkins/home/jobs/project/branches/master/workspace
            [Pipeline] {
            [Pipeline] sh
            [workspace] Running shell script
            + ./tools/test/run_unit_tests.sh
            Starting backend unit tests
            ============================= test session starts ==============================
            platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
            rootdir: /var/lib/jenkins/home/jobs/project/branches/master/workspace, inifile:
            plugins: asyncio-0.4.1, cov-2.3.0, mock-1.1
            collected 2 items

            src/tests/unit/magic.py ..

             generated xml file: /var/lib/jenkins/home/jobs/project/branches/master/workspace/reports/junit-unit.xml

            ----------- coverage: platform linux, python 3.5.2-final-0 -----------
            Coverage XML written to file reports/coverage-unit.xml

            =========================== 2 passed in 0.31 seconds ===========================
            Finished backend unit tests
            [Pipeline] step
            Recording test results
            {noformat}

            after that spinner spins forever
            This occurs both in parallel and in sequential runs.

            {code:java}
            stage 'Unit tests'
            node {
                sh './tools/test/run_unit_tests.sh'
                step([$class: 'JUnitResultArchiver', allowEmptyResults: true, testDataPublishers: [[$class: 'StabilityTestDataPublisher']], testResults: '**/reports/junit-unit.xml'])
            }
            {code}
            when executing following stage/step it hangs with last messages:


            {noformat}
            [Pipeline] stage (Unit tests)
            Entering stage Unit tests
            Proceeding
            [Pipeline] node
            Running on мастер in /var/lib/jenkins/home/jobs/project/branches/master/workspace
            [Pipeline] {
            [Pipeline] sh
            [workspace] Running shell script
            + ./tools/test/run_unit_tests.sh
            Starting backend unit tests
            ============================= test session starts ==============================
            platform linux -- Python 3.5.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
            rootdir: /var/lib/jenkins/home/jobs/project/branches/master/workspace, inifile:
            plugins: asyncio-0.4.1, cov-2.3.0, mock-1.1
            collected 2 items

            src/tests/unit/magic.py ..

             generated xml file: /var/lib/jenkins/home/jobs/project/branches/master/workspace/reports/junit-unit.xml

            ----------- coverage: platform linux, python 3.5.2-final-0 -----------
            Coverage XML written to file reports/coverage-unit.xml

            =========================== 2 passed in 0.31 seconds ===========================
            Finished backend unit tests
            [Pipeline] step
            Recording test results
            {noformat}

            after that spinner spins forever
            dcsobral Daniel Sobral added a comment -

            It stops hanging if you remove test stability.

            dcsobral Daniel Sobral added a comment - It stops hanging if you remove test stability.
            kamikaze Oleg Korsak added a comment -

            Yeah, but still this is a code from jenkins sample snippets. And actually removing functionality is not a FIX

            kamikaze Oleg Korsak added a comment - Yeah, but still this is a code from jenkins sample snippets. And actually removing functionality is not a FIX
            jamesdumay James Dumay made changes -
            Component/s workflow-plugin [ 18820 ]
            Component/s pipeline-stage-view-plugin [ 21139 ]
            Component/s pipeline-utility-steps-plugin [ 21135 ]
            Component/s pipeline-view-plugin [ 21435 ]

            The following hangs for me:

            step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml, **/target/failsafe-reports/TEST-*.xml'])
            

            Jenkins ver 2.10
            Pipeline / Pipeline: Build Step ver 2.2

            mscharp Michael Scharp added a comment - The following hangs for me: step([$class: 'JUnitResultArchiver' , testResults: '**/target/surefire-reports/TEST-*.xml, **/target/failsafe-reports/TEST-*.xml' ]) Jenkins ver 2.10 Pipeline / Pipeline: Build Step ver 2.2
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            eeaston Edward Easton added a comment - - edited

            I can reproduce this two ways - one using the old syntax and one with the new:

            New Syntax as per the snippet generator - this one hangs:

            stage('stage') {
                node {
                    sh "cp /tmp/junit*.xml ."    //  I have a junit file saved from a previous run here
                    junit testDataPublishers: [[$class: 'ClaimTestDataPublisher']], testResults: '**/junit*.xml'
                }
            }
            

            Old syntax - this one works OK

            stage('stage') {
                node {
                    sh "cp /tmp/junit*.xml ."    //  I have a junit file saved from a previous run here
                    step([$class: 'JUnitResultArchiver', testResults: '**/junit*.xml'])
                }
            }
            

            Here is the thread dump of the hung execution:

            Thread #6
            	at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#15])
            	at DSL.node(running on cn13)
            	at WorkflowScript.run(WorkflowScript:2)
            	at DSL.stage(Native Method)
            	at WorkflowScript.run(WorkflowScript:1)
            

            Versions:
            Jenkins - 2.7.4
            Pipeline - 2.4
            Pipeline Build Step - 2.2

            eeaston Edward Easton added a comment - - edited I can reproduce this two ways - one using the old syntax and one with the new: New Syntax as per the snippet generator - this one hangs: stage( 'stage' ) { node { sh "cp /tmp/junit*.xml ." // I have a junit file saved from a previous run here junit testDataPublishers: [[$class: 'ClaimTestDataPublisher' ]], testResults: '**/junit*.xml' } } Old syntax - this one works OK stage( 'stage' ) { node { sh "cp /tmp/junit*.xml ." // I have a junit file saved from a previous run here step([$class: 'JUnitResultArchiver' , testResults: '**/junit*.xml' ]) } } Here is the thread dump of the hung execution: Thread #6 at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#15]) at DSL.node(running on cn13) at WorkflowScript.run(WorkflowScript:2) at DSL.stage(Native Method) at WorkflowScript.run(WorkflowScript:1) Versions: Jenkins - 2.7.4 Pipeline - 2.4 Pipeline Build Step - 2.2

            I'm experiencing the same behavior as Edward here above.

               stage('Results') {
                  junit '**/target/surefire-reports/TEST-*.xml'  // --> WORKS FINE
            
                  junit testDataPublishers: [[$class: 'ClaimTestDataPublisher']], testResults: '**/target/surefire-reports/TEST-*.xml'  // --> HANGS
               }
            

            Thread dump:

            Thread #8
            	at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#13])
            	at WorkflowScript.run(WorkflowScript:20)
            	at DSL.stage(Native Method)
            	at WorkflowScript.run(WorkflowScript:19)
            	at DSL.node(running on )
            	at WorkflowScript.run(WorkflowScript:1)
            
            rickette Richard Kettelerij added a comment - I'm experiencing the same behavior as Edward here above. stage( 'Results' ) { junit '**/target/surefire-reports/TEST-*.xml' // --> WORKS FINE junit testDataPublishers: [[$class: 'ClaimTestDataPublisher' ]], testResults: '**/target/surefire-reports/TEST-*.xml' // --> HANGS } Thread dump: Thread #8 at DSL.step(hudson.tasks.junit.JUnitResultArchiver: running in thread: org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution [#13]) at WorkflowScript.run(WorkflowScript:20) at DSL.stage(Native Method) at WorkflowScript.run(WorkflowScript:19) at DSL.node(running on ) at WorkflowScript.run(WorkflowScript:1)
            rickette Richard Kettelerij made changes -
            Component/s junit-plugin [ 15499 ]

            eeaston I've traced down the issue we (apparently) both encountered and it was caused by the claim-plugin itself. I've contributed a fix to the claim-plugin to resolve this issue (https://github.com/jenkinsci/claim-plugin/pull/24). This fix is part of claim-plugin v2.9 which is already released. So if you update the claim-plugin your pipeline shouldn't hang on the ClaimTestDataPublisher as it did before.

            rickette Richard Kettelerij added a comment - eeaston I've traced down the issue we (apparently) both encountered and it was caused by the claim-plugin itself. I've contributed a fix to the claim-plugin to resolve this issue ( https://github.com/jenkinsci/claim-plugin/pull/24 ). This fix is part of claim-plugin v2.9 which is already released. So if you update the claim-plugin your pipeline shouldn't hang on the ClaimTestDataPublisher as it did before.
            elatt Erik Lattimore made changes -
            Component/s test-stability-plugin [ 17661 ]
            elatt Erik Lattimore made changes -
            Link This issue is duplicated by JENKINS-41901 [ JENKINS-41901 ]
            seanf Sean Flanigan made changes -
            Link This issue relates to JENKINS-33168 [ JENKINS-33168 ]
            seanf Sean Flanigan added a comment -

            The claim-plugin issue reported by eeaston and rickette seems to be JENKINS-39537.

            mscharp, you don't seem to be using Test Stability Plugin in that pipeline. I recommend you submit a separate bug report for junit-plugin (including logs) if you're still having that problem.

            kamikaze dcsobral Have you got any more details on the JUnitResultArchiver hang with StabilityTestDataPublisher? Did you have a lot of build history? Could it be JENKINS-33168?

            seanf Sean Flanigan added a comment - The claim-plugin issue reported by eeaston and rickette seems to be JENKINS-39537 . mscharp , you don't seem to be using Test Stability Plugin in that pipeline. I recommend you submit a separate bug report for junit-plugin (including logs) if you're still having that problem. kamikaze dcsobral Have you got any more details on the JUnitResultArchiver hang with StabilityTestDataPublisher? Did you have a lot of build history? Could it be JENKINS-33168 ?
            seanf Sean Flanigan made changes -
            Link This issue relates to JENKINS-42610 [ JENKINS-42610 ]
            jfstephe John Stephenson added a comment - - edited

            Adding this comment incase it helps shed some light on a possible solution or helps others wondering if this is still an issue.

            I also recently started getting this issue in an older version of Jenkins, upgraded (to Jenkins v2.73.2) and issue still present. I'm running 2 parallel builds but they run one after after another ATM as I've only one executor node. Both builds publish using the newer junit syntax. The second one consistently fails ATM.

            Changing this to the old syntax as suggested by Richard K makes it work as before.... for a bit ,but then it's not working now. Race condition?

             

            jfstephe John Stephenson added a comment - - edited Adding this comment incase it helps shed some light on a possible solution or helps others wondering if this is still an issue. I also recently started getting this issue in an older version of Jenkins, upgraded (to Jenkins v2.73.2) and issue still present. I'm running 2 parallel builds but they run one after after another ATM as I've only one executor node. Both builds publish using the newer junit syntax. The second one consistently fails ATM. Changing this to the old syntax as suggested by Richard K makes it work as before.... for a bit ,but then it's not working now. Race condition?  
            jamesdumay James Dumay made changes -
            Assignee Sam Van Oort [ svanoort ] Andrew Bayer [ abayer ]
            jamesdumay James Dumay made changes -
            Sprint Pipeline - Candidates [ 381 ]
            abayer Andrew Bayer made changes -
            Summary Pipeline hangs on JUnitResultArchiver step Pipeline hangs on JUnitResultArchiver step with test publishers
            vivek Vivek Pandey made changes -
            Sprint Pipeline - Candidates [ 381 ]
            vivek Vivek Pandey made changes -
            Rank Ranked higher
            nfalco Nikolas Falco added a comment - - edited

            We have the same issue where in a pipeline we run in parallel two maven build that perform integration tests on different application server. We would like collect the tests distinguishing (because are the same) them by branch. When maven build in all branches ends the pipeline hang during the report generation I suppose.

             

            We had try both:

            junit testDataPublishers: [[$class: 'JUnitFlakyTestDataPublisher']], testResults: '**/target/surefire-reports/*.xml'

            and

            junit '**/target/surefire-reports/*.xml'
            nfalco Nikolas Falco added a comment - - edited We have the same issue where in a pipeline we run in parallel two maven build that perform integration tests on different application server. We would like collect the tests distinguishing (because are the same) them by branch. When maven build in all branches ends the pipeline hang during the report generation I suppose.   We had try both: junit testDataPublishers: [[$class: 'JUnitFlakyTestDataPublisher' ]], testResults: '**/target/surefire-reports/*.xml' and junit '**/target/surefire-reports/*.xml'
            vivek Vivek Pandey made changes -
            Labels 2.0 junit pipeline pipeline-build-step-plugin 2.0 junit pipeline pipeline-build-step-plugin triaged-2018-11
            nfalco Nikolas Falco made changes -
            Link This issue is duplicated by JENKINS-37443 [ JENKINS-37443 ]
            abayer Andrew Bayer made changes -
            Assignee Andrew Bayer [ abayer ]

            People

              Unassigned Unassigned
              kamikaze Oleg Korsak
              Votes:
              9 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated: