Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68061

On parallel execution stages are not marked according to threshold values

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • xunit-plugin
    • None
    • 3.0.6

      We use xunit in a parallel pipeline to analyze test report files.

      When there are multiple test failures, xunit only marks the first calling stage as unstable. All other stages are marked as passed.

      Here is an stage overview of a MWE illustrating the problem:

      All three stages perform the xunit analysis on the same test report file which contains one failing test. The xunit call is like so:

      xunit thresholds: [failed(unstableThreshold: '0')],
                                      tools: [GoogleTest(deleteOutputFiles: true,
                                          failIfNotNew: false,
                                          pattern: "*.xml",
                                          skipNoTestFiles: false,
                                          stopProcessingIfError: true)]
      

      The only difference between the stages is a delay before calling xunit which has been swapped between the three stages for each run.

      I would expect all three stages to be marked unstable, like so:

       


       

      We use xunit version 3.0.5,  and Jenkins Core 2.319.3

          [JENKINS-68061] On parallel execution stages are not marked according to threshold values

          O B created issue -

          Nikolas Falco added a comment -

          Could you provide me the piece of pipeline job are you using to run the parallel block. I mean the code that runs the stages no1, no2, no3.

          Could you also attach the report file you are using?

          Nikolas Falco added a comment - Could you provide me the piece of pipeline job are you using to run the parallel block. I mean the code that runs the stages no1, no2, no3. Could you also attach the report file you are using?
          O B made changes -
          Attachment New: unstable_test.xml [ 57482 ]
          Attachment New: unstable_test.groovy [ 57483 ]

          O B added a comment -

          Sure thing, I attached the groovy script and test.xml

          O B added a comment - Sure thing, I attached the groovy script and test.xml
          O B made changes -
          Description Original: We use xunit in a parallel pipeline to analyze test report files.

          When there are multiple test failures, xunit only marks the first calling stage as unstable. All other stages are marked as passed.

          Here is an stage overview of a MWE illustrating the problem:
          !image-2022-03-17-11-38-41-869.png!

          All three stages perform the xunit analysis on the same test report file which contains one failing test. The xunit call is like so:
          {code:groovy}
          xunit thresholds: [failed(unstableThreshold: '0')],
                                          tools: [GoogleTest(deleteOutputFiles: true,
                                              failIfNotNew: false,
                                              pattern: "*.xml",
                                              skipNoTestFiles: false,
                                              stopProcessingIfError: true)]
          {code}
          The only difference between the stages is a delay before calling xunit which has been swapped between the three stages for each run.

          I would expect all three stages to be marked unstable, like so:
           !image-2022-03-17-11-45-06-156.png|thumbnail!

          New: We use xunit in a parallel pipeline to analyze test report files.

          When there are multiple test failures, xunit only marks the first calling stage as unstable. All other stages are marked as passed.

          Here is an stage overview of a MWE illustrating the problem:
           !image-2022-03-17-11-38-41-869.png!

          All three stages perform the xunit analysis on the same test report file which contains one failing test. The xunit call is like so:
          {code:groovy}
          xunit thresholds: [failed(unstableThreshold: '0')],
                                          tools: [GoogleTest(deleteOutputFiles: true,
                                              failIfNotNew: false,
                                              pattern: "*.xml",
                                              skipNoTestFiles: false,
                                              stopProcessingIfError: true)]
          {code}
          The only difference between the stages is a delay before calling xunit which has been swapped between the three stages for each run.

          I would expect all three stages to be marked unstable, like so:
           !image-2022-03-17-11-45-06-156.png|thumbnail!

           
          ----
           

          We use xunit version 3.0.5,  and Jenkins Core 2.319.3

          Nikolas Falco added a comment -

          Could you look me if global build test result is correct? I mean in that case it should report 3 failures.

          Nikolas Falco added a comment - Could you look me if global build test result is correct? I mean in that case it should report 3 failures.

          Nikolas Falco added a comment -

          I got the issue. If the first stage mark build UNSTABLE the second one since the build is not worst than UNSTABLE do not do anything so it appear green.

          Nikolas Falco added a comment - I got the issue. If the first stage mark build UNSTABLE the second one since the build is not worst than UNSTABLE do not do anything so it appear green.
          Nikolas Falco made changes -
          Issue Type Original: Improvement [ 4 ] New: Bug [ 1 ]
          Nikolas Falco made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Nikolas Falco made changes -
          Summary Original: Set all stages according to threshold values New: On parallel execution stages are not marked according to threshold values

            nfalco Nikolas Falco
            keksmaster O B
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: