Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55580

Inconsistent build result for pipeline jobs that have parallel cucumber tests

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • jenkins 2.121.3, cucumber test-results plugin 0.10.1

      As the plugin processes test results from the phases, it merges the results together. If there are passed scenarios and failed, the overall job is marked as unstable, if all jobs fail then it is marked as failure.

       

      The inconsistency is:

      a) if the first cucumber test to finish fails and subsequent tests pass, the overall pipeline turns red.

      vs

      b) if the first cucumber test passes and subsequent ones fail, the pipeline turns yellow.

       

      It is desirable that the result of the pipeline be consistent, regardless of the order the tests finish in.

       

      Alternatively, if we could pass in a parameter to avoid any manipulation of the overall build result from within this plugin, we can handle manipulating the build result as part of a separate phase after the tests complete.

          [JENKINS-55580] Inconsistent build result for pipeline jobs that have parallel cucumber tests

          James Nord added a comment -

          what version of the junit plugin are you using?

          James Nord added a comment - what version of the junit plugin are you using?

          Dustin Neray added a comment -

          junit version 1.24

          Dustin Neray added a comment - junit version 1.24

          Dustin Neray added a comment -

          Assuming I'm looking in the right place, it looks like the issue is here: https://github.com/jenkinsci/cucumber-testresult-plugin/blob/master/src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberTestResultArchiver.java where it changes the build result as tests finish. After the first test fails, it sets the overall result to failure since action.getResult().getTotalCount() == action.getResult().getFailCount() will be true

          Dustin Neray added a comment - Assuming I'm looking in the right place, it looks like the issue is here: https://github.com/jenkinsci/cucumber-testresult-plugin/blob/master/src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberTestResultArchiver.java  where it changes the build result as tests finish. After the first test fails, it sets the overall result to failure since action.getResult().getTotalCount() == action.getResult().getFailCount() will be true

          James Nord added a comment -

          that would be correct. are you only running one scenario for the invocation of the recorder, as all the tests would need to fail for the first report which is not very common and would be more indicative of a setup failure than test failure (hence that code) 

          James Nord added a comment - that would be correct. are you only running one scenario for the invocation of the recorder, as all the tests would need to fail for the first report which is not very common and would be more indicative of a setup failure than test failure (hence that code) 

          Dustin Neray added a comment -

          Yes, each individual scenario runs in parallel since these are (unfortunately) quite long running tests

          Dustin Neray added a comment - Yes, each individual scenario runs in parallel since these are (unfortunately) quite long running tests

            Unassigned Unassigned
            dneray Dustin Neray
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: