Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56502

xunit thresholds may have a bug in parallel pipeline runs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: xunit-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

      i think that it is better not to abuse old closed issue with this one.

      here is the summary of discussion i had with Nicholas.:

      ======================================

      i see that xunit thresholds produce non deterministic results in parallel pipe runs 

      This is how i define xunit thresholds:

       

      xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                      tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                              deleteOutputFiles: true,
                              failIfNotNew: true,
                              pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                              skipNoTestFiles: true,
                              stopProcessingIfError: true)]
      

       I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

       

      I have some build new failed tests:

      Test Result (40 failures / +17)

        i expect to get failed status, but xunit still sets the build to be unstable:

      01:57:59.736  INFO: Starting to record.
      01:57:59.736  INFO: Processing Custom Tool
      01:58:00.146  INFO: Stopping recording.
      01:58:00.147  INFO: Check 'Failed Tests' threshold.
      01:58:00.147  INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
      

       I use:

      some more details:

      I want to define following behavior:

      Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. 

      This is how I use Snippet Generator  

       

      this is what it produces: 

      xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
      

      Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

       

      Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

       That doesn't make sense to me:

      Failed thresholds are analysed before unstable thresholds: that's ok, in example above only was supposed to kick-in, i hope the logic says that if one threshold declared failure, later threshold cannot cancel it.  

      Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

      So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe

        Attachments

          Activity

          vladichko Vlad Aginsky created issue -
          vladichko Vlad Aginsky made changes -
          Field Original Value New Value
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

           On this Nicolas kindly replied:

          Anyway if this is your configuration (it's not clear from your post, seems commented)
          {noformat}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
          {noformat}
          the unstable build status is correct and is in accordance with what the tooltips document.

           
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

           On this Nicolas kindly replied:

          Anyway if this is your configuration (it's not clear from your post, seems commented)
          {noformat}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
          {noformat}
          the unstable build status is correct and is in accordance with what the tooltips document.

           
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=646,height=150!

          this is what it produces:

           
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=646,height=150!

          this is what it produces:

           
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=534,height=124!

          this is what it produces:

           
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=534,height=124!

          this is what it produces:

           
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=676,height=157!

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          vladichko Vlad Aginsky made changes -
          Attachment image-2019-03-04-09-13-18-589.png [ 46358 ]
          vladichko Vlad Aginsky made changes -
          Attachment image-2019-03-11-14-46-28-394.png [ 46359 ]
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-04-09-13-18-589.png|width=676,height=157!

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           

          Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

           

          and that doesn't make sense to me:

          *Failed thresholds are analysed before unstable thresholds—> thats ok, in example above only fail was supposed to kick-in

          *Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

          so after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be relates to some race between  parallel parts of pipe
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           

          Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

           

          and that doesn't make sense to me:

          *Failed thresholds are analysed before unstable thresholds—> thats ok, in example above only fail was supposed to kick-in

          *Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

          so after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be relates to some race between  parallel parts of pipe
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           

          Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

           That doesn't make sense to me:

          _*Failed thresholds are analysed before unstable thresholds:*_ thats ok, in example above only fail was supposed to kick-in

          _*Total count of test failure are taken into account before of New test failures:*_ I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

          So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe
          vladichko Vlad Aginsky made changes -
          Description This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           

          Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

           That doesn't make sense to me:

          _*Failed thresholds are analysed before unstable thresholds:*_ thats ok, in example above only fail was supposed to kick-in

          _*Total count of test failure are taken into account before of New test failures:*_ I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

          So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe
          This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

          i think that it is better not to abuse old closed issue with this one.

          here is the summary of discussion i had with Nicholas.:

          ======================================

          i see that xunit thresholds produce non deterministic results in parallel pipe runs 

          This is how i define xunit thresholds:

           
          {code:java}
          xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                          tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                                  deleteOutputFiles: true,
                                  failIfNotNew: true,
                                  pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                                  skipNoTestFiles: true,
                                  stopProcessingIfError: true)]
          {code}
           I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

           

          I have some build new failed tests:
          |!https://swdvp-jnk01.anpa.corp.amazon.com:9080/static/da57df94/images/48x48/clipboard.png!|Test Result (40 failures / +17)|

            i expect to get failed status, but xunit still sets the build to be unstable:
          {code:java}
          01:57:59.736 INFO: Starting to record.
          01:57:59.736 INFO: Processing Custom Tool
          01:58:00.146 INFO: Stopping recording.
          01:58:00.147 INFO: Check 'Failed Tests' threshold.
          01:58:00.147 INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
          {code}
           I use:
           *  [Jenkins ver. 2.160|https://jenkins.io/]
           *  xUnit plugin 2.3.2

          some more details:

          I want to define following behavior:

          {{_Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status._ }}

          This is how I use Snippet Generator  

          !image-2019-03-11-14-46-28-394.png|width=568,height=132!

           

          this is what it produces: 
          {code:java}
          xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
          {code}
          *Important note:* i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

           

          Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

           That doesn't make sense to me:

          _*Failed thresholds are analysed before unstable thresholds:*_ that's ok, in example above only was supposed to kick-in, i hope the logic says that if one threshold declared failure, later threshold cannot cancel it.  

          _*Total count of test failure are taken into account before of New test failures:*_ I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

          So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe
          nfalco Nikolas Falco made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Closed [ 6 ]

            People

            Assignee:
            nfalco Nikolas Falco
            Reporter:
            vladichko Vlad Aginsky
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: