Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65414

Tests failing to suite teardown are evaluated as passed in robot publisher

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • robot-plugin
    • None
    • Robot Framework Plugin 3.0.0
      Robot Framework 4.0.0 / 4.0.1

      Tests that pass execution but fail at suite teardown phase are displayed in the 'Passed' column in 'Robot Test Summary' (while the robot log itself does display the results correctly). 

      This also results in the build being evaluated as success in case there are only suite teardown failures and robot is executed with --nostatusrc while publisher is used to determine the build status.

          [JENKINS-65414] Tests failing to suite teardown are evaluated as passed in robot publisher

          Aleksi Simell added a comment - - edited

          Hi paiviatvarjo

          Have you unchecked the criticality checkbox or do you use onlyCritical: false in your pipeline as explained in the plugin documentation?

          EDIT: I just re-read the issue and looks like I made some hasty assumptions. This could actually be a defect, but can you still verify the question above?

          Aleksi Simell added a comment - - edited Hi paiviatvarjo Have you unchecked the criticality checkbox or do you use onlyCritical: false in your pipeline as explained in the plugin documentation ? EDIT: I just re-read the issue and looks like I made some hasty assumptions. This could actually be a defect, but can you still verify the question above?

          Yes, and according to the documentation it also defaults to false?

          Here's what we use:
          step (
          [
          $class : 'RobotPublisher',
          outputPath : outputPath,
          outputFileName : outputFileName,
          reportFileName : reportFileName,
          logFileName : logFileName,
          disableArchiveOutput : disableArchiveOutput,
          passThreshold : passThreshold,
          unstableThreshold : unstableThreshold,
          otherFiles : otherFiles,
          onlyCritical : false
          ]
          )

          Päivi Karjalainen added a comment - Yes, and according to the documentation it also defaults to false? Here's what we use: step ( [ $class : 'RobotPublisher', outputPath : outputPath, outputFileName : outputFileName, reportFileName : reportFileName, logFileName : logFileName, disableArchiveOutput : disableArchiveOutput, passThreshold : passThreshold, unstableThreshold : unstableThreshold, otherFiles : otherFiles, onlyCritical : false ] )

          Aleksi Simell added a comment -

          Thanks, I'll see if I can reproduce this and come back to this.

          Aleksi Simell added a comment - Thanks, I'll see if I can reproduce this and come back to this.

          Just for reference, here's an example suite I can reproduce with:

          *** Settings ***
          Test Setup    Log    Test Setup
          Test Teardown    Log    Test Teardown
          
          Suite Teardown    Fail   Boom
          
          *** Test Cases ***
          Foo Test
              Log    foo
          
          Foo Test 2
              Log    foo2
          
          *** Keywords ***
          

          Päivi Karjalainen added a comment - Just for reference, here's an example suite I can reproduce with: *** Settings *** Test Setup Log Test Setup Test Teardown Log Test Teardown Suite Teardown Fail Boom *** Test Cases *** Foo Test Log foo Foo Test 2 Log foo2 *** Keywords ***

          Aleksi Simell added a comment -

          https://github.com/jenkinsci/robot-plugin/pull/42 Created to fix this issue.

          Root cause: Checking teardown status was case sensitive.

          Fix: Make it case insensitive to support both RF 3.x and 4.x.

          Aleksi Simell added a comment - https://github.com/jenkinsci/robot-plugin/pull/42 Created to fix this issue. Root cause: Checking teardown status was case sensitive. Fix: Make it case insensitive to support both RF 3.x and 4.x.

          Excellent, thank you!

          Päivi Karjalainen added a comment - Excellent, thank you!

          Aleksi Simell added a comment -

          PR merged. Will be available in next release.

          Aleksi Simell added a comment - PR merged. Will be available in next release.

          Aleksi Simell added a comment -

          Fix was released in version 3.0.1. Will be available in update center within a few hours.

          Aleksi Simell added a comment - Fix was released in version 3.0.1. Will be available in update center within a few hours.

            aleksisimell Aleksi Simell
            paiviatvarjo Päivi Karjalainen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: