Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56502

xunit thresholds may have a bug in parallel pipeline runs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • xunit-plugin
    • None

      This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

      i think that it is better not to abuse old closed issue with this one.

      here is the summary of discussion i had with Nicholas.:

      ======================================

      i see that xunit thresholds produce non deterministic results in parallel pipe runs 

      This is how i define xunit thresholds:

       

      xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                      tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                              deleteOutputFiles: true,
                              failIfNotNew: true,
                              pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                              skipNoTestFiles: true,
                              stopProcessingIfError: true)]
      

       I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

       

      I have some build new failed tests:

      Test Result (40 failures / +17)

        i expect to get failed status, but xunit still sets the build to be unstable:

      01:57:59.736  INFO: Starting to record.
      01:57:59.736  INFO: Processing Custom Tool
      01:58:00.146  INFO: Stopping recording.
      01:58:00.147  INFO: Check 'Failed Tests' threshold.
      01:58:00.147  INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
      

       I use:

      some more details:

      I want to define following behavior:

      Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. 

      This is how I use Snippet Generator  

       

      this is what it produces: 

      xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
      

      Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

       

      Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

       That doesn't make sense to me:

      Failed thresholds are analysed before unstable thresholds: that's ok, in example above only was supposed to kick-in, i hope the logic says that if one threshold declared failure, later threshold cannot cancel it.  

      Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

      So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe

            nfalco Nikolas Falco
            vladichko Vlad Aginsky
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: