Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35911

CTest returning non-zero error code should not stop the build

    XMLWordPrintable

Details

    Description

      If a test fails then CTest returns a non-zero return code (8 in my configuration) and jenkins build is stopped.
      This makes the build fail even if higher unstable and fail thresholds are selected.
      An option to select the return code to be ignored will work the best.

      Attachments

        Activity

          15knots Martin Weber added a comment -

          Would it be sufficient to add a boolean option that allows to ignore the exit status of ctest it it does not indicate success?

          15knots Martin Weber added a comment - Would it be sufficient to add a boolean option that allows to ignore the exit status of ctest it it does not indicate success?
          dcarp Dragos Carp added a comment - - edited

          Yes, adding an option to ignore the exit status would do the job in my case.
          On the other side, it can be that CTest returns a different error code if there is another kind of error (wrong parameters, etc.). That kind of errors shouldn't be ignored. Unfortunately the CTest documentation does not specifies the error codes.

          dcarp Dragos Carp added a comment - - edited Yes, adding an option to ignore the exit status would do the job in my case. On the other side, it can be that CTest returns a different error code if there is another kind of error (wrong parameters, etc.). That kind of errors shouldn't be ignored. Unfortunately the CTest documentation does not specifies the error codes.
          dcarp Dragos Carp added a comment -

          I have reported the issue to CTest: https://gitlab.kitware.com/cmake/cmake/issues/16151

          dcarp Dragos Carp added a comment - I have reported the issue to CTest: https://gitlab.kitware.com/cmake/cmake/issues/16151
          15knots Martin Weber added a comment -

          Please try https://jenkins.ci.cloudbees.com/job/plugins/job/cmakebuilder-plugin/org.jenkins-ci.plugins$cmakebuilder/102/artifact/org.jenkins-ci.plugins/cmakebuilder/2.4.3-SNAPSHOT/cmakebuilder-2.4.3-SNAPSHOT.hpi and provide some feedback.
          My unittests are unreliable here, since ctest v. 2.8.11.2 seems to always exit with success so they never fail for this issue .
          And please have a look at the online help and send suggestions, as I am not a native speaker.

          15knots Martin Weber added a comment - Please try https://jenkins.ci.cloudbees.com/job/plugins/job/cmakebuilder-plugin/org.jenkins-ci.plugins$cmakebuilder/102/artifact/org.jenkins-ci.plugins/cmakebuilder/2.4.3-SNAPSHOT/cmakebuilder-2.4.3-SNAPSHOT.hpi and provide some feedback. My unittests are unreliable here, since ctest v. 2.8.11.2 seems to always exit with success so they never fail for this issue . And please have a look at the online help and send suggestions, as I am not a native speaker.
          dcarp Dragos Carp added a comment - - edited

          Unfortunately it didn't work out as expected. Actually even that the Plugin Manager shows that "CMake plugin 2.4.3-SNAPSHOT (private-e95ede2f-jenkins)" is installed, I couldn't find any option to disable the checking of the exit code. Currently I'm using Jenkins 2.5 installed on Windows 2008 Server.
          The relevant section in the "Console Output":

          The following tests FAILED:
          	 72 - testFoo (Failed)
          	 74 - testBar (Failed)
          	 75 - testBaz (Failed)
          	116 - testQux (SEGFAULT)
          
          Errors while running CTest
          Build step 'CMake/CPack/CTest execution' changed build result to FAILURE
          
          dcarp Dragos Carp added a comment - - edited Unfortunately it didn't work out as expected. Actually even that the Plugin Manager shows that "CMake plugin 2.4.3-SNAPSHOT (private-e95ede2f-jenkins)" is installed, I couldn't find any option to disable the checking of the exit code. Currently I'm using Jenkins 2.5 installed on Windows 2008 Server. The relevant section in the "Console Output": The following tests FAILED: 72 - testFoo (Failed) 74 - testBar (Failed) 75 - testBaz (Failed) 116 - testQux (SEGFAULT) Errors while running CTest Build step 'CMake/CPack/CTest execution' changed build result to FAILURE
          15knots Martin Weber added a comment -

          You will find a new text box labeled Ignore exit codes in the CMake/CPack/CTest build step now, which should satisfy your needs.

          15knots Martin Weber added a comment - You will find a new text box labeled Ignore exit codes in the CMake/CPack/CTest build step now, which should satisfy your needs.
          dcarp Dragos Carp added a comment -

          Today I got the chance to test this feature in version 2.4.3 and it works as expected. Thank you!

          dcarp Dragos Carp added a comment - Today I got the chance to test this feature in version 2.4.3 and it works as expected. Thank you!

          People

            15knots Martin Weber
            dcarp Dragos Carp
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: