Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56502

xunit thresholds may have a bug in parallel pipeline runs

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • xunit-plugin
    • None

      This is spin-off issue from https://issues.jenkins-ci.org/browse/JENKINS-54255

      i think that it is better not to abuse old closed issue with this one.

      here is the summary of discussion i had with Nicholas.:

      ======================================

      i see that xunit thresholds produce non deterministic results in parallel pipe runs 

      This is how i define xunit thresholds:

       

      xunit thresholds: [failed(unstableThreshold: '0',failureNewThreshold: '0')],
                      tools: [Custom(customXSL: "${JENKINS_URL}/userContent/pytest-to-junit.xsl",
                              deleteOutputFiles: true,
                              failIfNotNew: true,
                              pattern: "${env.DOWNLOADED_ARTIFACTS_PATH}/${env.TEST_RESULTS_PATH}/*.xml",
                              skipNoTestFiles: true,
                              stopProcessingIfError: true)]
      

       I run few parallel branches of jenkins pipe, each in its own workspace and each pipe branch uploads it test results independently.

       

      I have some build new failed tests:

      Test Result (40 failures / +17)

        i expect to get failed status, but xunit still sets the build to be unstable:

      01:57:59.736  INFO: Starting to record.
      01:57:59.736  INFO: Processing Custom Tool
      01:58:00.146  INFO: Stopping recording.
      01:58:00.147  INFO: Check 'Failed Tests' threshold.
      01:58:00.147  INFO: The new number of tests for the threshold 'Failed Tests' exceeds the specified "new failure threshold" value.
      

       I use:

      some more details:

      I want to define following behavior:

      Any new failed test to fail the build, if no new failures, any old failure to cause to unstable build status. 

      This is how I use Snippet Generator  

       

      this is what it produces: 

      xunit thresholds: [failed(failureNewThreshold: '0', unstableThreshold: '0')], tools: []
      

      Important note: i see that in some cases it behaves just right, but sometimes it produces yellow status on new failed tests, just as described above.

       

      Button line of previous discussion with Nicolas was:Failed thresholds are analysed before unstable thresholds. Total count of test failure are taken into account before of New test failures. This logic is applied to all thresholds (skipped, failed and passed). This means your scenario is not possible.

       That doesn't make sense to me:

      Failed thresholds are analysed before unstable thresholds: that's ok, in example above only was supposed to kick-in, i hope the logic says that if one threshold declared failure, later threshold cannot cancel it.  

      Total count of test failure are taken into account before of New test failures: I hope that means no matter how many total failure there are, in my case new failures will define the outcome.

      So, after all, it seems that there is a problem with my results, and again, as i wrote above, in most cases it behaves ok, may be related to some race between  parallel parts of pipe

          [JENKINS-56502] xunit thresholds may have a bug in parallel pipeline runs

          Nikolas Falco added a comment - - edited

          If for parallel run you mean a multi-branch pipeline project where each job has an own jenkins job and (as you describe) workspace and so on... that this is not a parallel run. It has no issue in that way. If you mean that there is just a jenkins job but you can cun concurrent build and each build has an own workspace than this could be a problem because the xunit plugin takes the total count of previous build that is stored at job level (not workspace) and all depends from who build run xunit plugin as first. I think last case is not your case because you describe run different branch and you had not speak about concurrent build of same job. This to say that parallel is not the root cause.

          To understand better the issue I need the test trend result (latest 10 build are sufficient) where the issue happens.
          Could you also give to me the job definition (whick kind of job is)

          Nikolas Falco added a comment - - edited If for parallel run you mean a multi-branch pipeline project where each job has an own jenkins job and (as you describe) workspace and so on... that this is not a parallel run. It has no issue in that way. If you mean that there is just a jenkins job but you can cun concurrent build and each build has an own workspace than this could be a problem because the xunit plugin takes the total count of previous build that is stored at job level (not workspace) and all depends from who build run xunit plugin as first. I think last case is not your case because you describe run different branch and you had not speak about concurrent build of same job. This to say that parallel is not the root cause. To understand better the issue I need the test trend result (latest 10 build are sufficient) where the issue happens. Could you also give to me the job definition (whick kind of job is)

            nfalco Nikolas Falco
            vladichko Vlad Aginsky
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: