Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46553

Maven surefire timeout treated as successful build

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • maven-plugin
    • None
    • linux x64, Jenkins LTS 2.60.3, Maven Integration plugin 2.17

      If a project configures surefire to timeout after a reasonable time using  -Dsurefire.exitTimeout=180 -Dsurefire.timeout=180 and the tests time out you'll see

      [ERROR] There was a timeout or other error in the fork
      [JENKINS] Recording test results

      However Jenkins decides that the tests succeeded and completes the build and all success actions.

      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------

      This condition can be simulated by setting an artificially low surefire timeout.

          [JENKINS-46553] Maven surefire timeout treated as successful build

          Falko Modler added a comment - - edited

          Same here. In a regular shell (locally, not in Jenkins) the error is manifested as a build failure:

          [INFO]
          [INFO] Results:
          [INFO]
          [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 57.572 s
          [INFO] Finished at: 2019-09-09T13:36:55+02:00
          [INFO] Final Memory: 50M/1024M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
                  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
                  at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
                  at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
                  at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
                  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
                  at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
                  at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
                  at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
                  at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
                  at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
                  at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
          Caused by: org.apache.maven.plugin.MojoFailureException: There was a timeout or other error in the fork
                  at org.apache.maven.plugin.surefire.SurefireHelper.throwException(SurefireHelper.java:240)
                  at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:112)
                  at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:354)
                  at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1008)
                  at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:854)
                  at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
                  at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
                  ... 20 more
          [ERROR]
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          

          That is with mvn -e .... Without it you won't see the stacktrace but the return code is 1 either way!

          Falko Modler added a comment - - edited Same here. In a regular shell (locally, not in Jenkins) the error is manifested as a build failure: [INFO] [INFO] Results: [INFO] [INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 57.572 s [INFO] Finished at: 2019-09-09T13:36:55+02:00 [INFO] Final Memory: 50M/1024M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project some-project: There was a timeout or other error in the fork at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoFailureException: There was a timeout or other error in the fork at org.apache.maven.plugin.surefire.SurefireHelper.throwException(SurefireHelper.java:240) at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:112) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:354) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1008) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:854) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException That is with mvn -e ... . Without it you won't see the stacktrace but the return code is 1 either way!

          Falko Modler added a comment - - edited

          Plot twist: I don't think anymore that Jenkins is to blame, at least in my case:

          I am (and probably most other Jenkins users are) setting -Dmaven.test.failure.ignore=true to get all test results instead of a failing build after the first module with test failures.
          This is where https://issues.apache.org/jira/browse/SUREFIRE-953 comes into play!

          What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts.

          For now I am using the following pipeline lib function which I just need to call in my pipelines (in some post block) without any parameters:

          failOnKnownConsoleLogPatterns.groovy
          // Parses the console log file for known errors (that Jenkins does not properly recognize as build failures) and fails the build if any are found.
          def call() {
              def known = [
                  'There was a timeout or other error in the fork'    // https://issues.jenkins-ci.org/browse/JENKINS-46553
              ]
              def consoleLogFile = "${WORKSPACE}/../builds/${BUILD_NUMBER}/log"
              println "Checking for known errors in console log file that are not properly handled by Jenkins: ${consoleLogFile}"
              // grep the cosole log, notes:
              //   - "set +x" to avoid echoing the grep command which would lead to a false positive
              //   - "set +e" to avoid build failure due to non-zero exit code (otherwise "exit 0" wouldn't be reached)
              def foundErrors = sh(script: "set +xe && grep '${known.join('|')}' ${consoleLogFile}; exit 0", returnStdout: true)
              if (foundErrors != "") {
                  error("Found known errors in the log (see previous outputs for more context):\n${foundErrors}")
              }
          }
          

          PS: Yes I know, directly "grepping" the log file is not good practise. I also haven't tested this with/on slaves.
          I do also know that there are log parsing plugins but they do come with their own litte quirks, bugs and overhead...

          Falko Modler added a comment - - edited Plot twist : I don't think anymore that Jenkins is to blame, at least in my case: I am (and probably most other Jenkins users are) setting -Dmaven.test.failure.ignore=true to get all test results instead of a failing build after the first module with test failures. This is where https://issues.apache.org/jira/browse/SUREFIRE-953 comes into play! What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts. For now I am using the following pipeline lib function which I just need to call in my pipelines (in some post block) without any parameters: failOnKnownConsoleLogPatterns.groovy // Parses the console log file for known errors (that Jenkins does not properly recognize as build failures) and fails the build if any are found. def call() { def known = [ 'There was a timeout or other error in the fork' // https://issues.jenkins-ci.org/browse/JENKINS-46553 ] def consoleLogFile = "${WORKSPACE}/../builds/${BUILD_NUMBER}/log" println "Checking for known errors in console log file that are not properly handled by Jenkins: ${consoleLogFile}" // grep the cosole log, notes: // - "set +x" to avoid echoing the grep command which would lead to a false positive // - "set +e" to avoid build failure due to non-zero exit code (otherwise "exit 0" wouldn't be reached) def foundErrors = sh(script: "set +xe && grep '${known.join(' | ')}' ${consoleLogFile}; exit 0" , returnStdout: true ) if (foundErrors != "") { error( "Found known errors in the log (see previous outputs for more context):\n${foundErrors}" ) } } PS: Yes I know, directly "grepping" the log file is not good practise. I also haven't tested this with/on slaves. I do also know that there are log parsing plugins but they do come with their own litte quirks, bugs and overhead...

          Falko Modler added a comment -

          What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts.

          I've just created a ticket for that: https://issues.apache.org/jira/browse/SUREFIRE-1728

          Falko Modler added a comment - What we would need is another Maven Surefire property/feature, that differentiates between regular test failures and timeouts. I've just created a ticket for that: https://issues.apache.org/jira/browse/SUREFIRE-1728

          Tibor Digana added a comment - - edited

          bondolo

          famod

          We cannot modify the purpose of maven.test.failure.ignore .

          Let's explain it: The test set will end up with Result Set (total, failures, errors, flakes, skipped) and the test frameworks (JUnit, TestNG, etc) provide RunListener sending the test events into the final Result Set. All the teminology you can see in {{maven.test.failure.ignore}} talks about the failures and it means the Result Set. It has nothing to do with any timeouts and vice versa!

          If we have to fix this without touching the TXT and XML reports, I am all the ear how.

          My proposal is to introduce surefire.failIfTimeout=true and failsafe.failIfTimeout=true. It is complemetary principle to the parameter that we already have with failIfNoTests=true. So we control the behavior on fine grained level.

          Setting maven.test.failure.ignore=true would not bypass the failIfNoTests=true and the build fails anyway with not having any tests in the project. The same should be with the timeouts. The only difference are the default values, where failIfNotests has false, and failIfTimeout would have true.

          If the user wants to avoid build failure due to the plugin, run the cmd mvn test -Dmaven.test.failure.ignore=true -Dsurefire.failIfTimeout=false

           

          If the user wants to avoid test failure(s) run the cmd mvn test -Dmaven.test.failure.ignore=true

           

          Feel free to comment on this.

           

          Tibor Digana added a comment - - edited bondolo famod We cannot modify the purpose of maven.test.failure.ignore . Let's explain it: The test set will end up with Result Set (total, failures, errors, flakes, skipped) and the test frameworks (JUnit, TestNG, etc) provide RunListener sending the test events into the final Result Set. All the teminology you can see in {{maven.test.failure.ignore }} talks about the failures and it means the Result Set. It has nothing to do with any timeouts and vice versa! If we have to fix this without touching the TXT and XML reports, I am all the ear how. My proposal is to introduce surefire.failIfTimeout=true and failsafe.failIfTimeout=true . It is complemetary principle to the parameter that we already have with failIfNoTests=true . So we control the behavior on fine grained level. Setting maven.test.failure.ignore =true would not bypass the failIfNoTests=true and the build fails anyway with not having any tests in the project. The same should be with the timeouts. The only difference are the default values, where failIfNotests has false , and failIfTimeout would have true . If the user wants to avoid build failure due to the plugin, run the cmd mvn test -Dmaven.test.failure.ignore=true -Dsurefire.failIfTimeout=false   If the user wants to avoid test failure(s) run the cmd mvn test -Dmaven.test.failure.ignore=true   Feel free to comment on this.  

          Here's my workaround for scriped pipeline. It requires some script security exemptions but otherwise works quite nicely. I'm also trying to print out which test class had the issue.

          def detectedErrors = anyErrorDetected()
          if(detectedErrors){
            error detectedErrors
          }
          
          
          @com.cloudbees.groovy.cps.NonCPS
          def anyErrorDetected() {
            logReader = currentBuild.rawBuild.logReader
            def startedNotFinished = []
            def forkTimeout = false
            def otherForkIssue = false
            logReader.eachLine {
              def startMatcher = it =~ /.*Running (?<name>[\w.-]+)/
              def endMatcher = it =~ /.*Tests run.* in (?<name>[\w.-]+)/
              if(startMatcher.matches()){
                startedNotFinished.add(startMatcher.group("name"))
              }
              if(endMatcher.matches()){
                startedNotFinished.remove(endMatcher.group("name"))
              }
          
              if(it =~ /There was a timeout or other error in the fork/){
                forkTimeout = true
              } else if (it =~ /Error occurred in starting fork/){
                otherForkIssue = true
              }
            }
          
            if(forkTimeout){
              return "There was a timeout or other error in the fork. Potential tests with issue: " + startedNotFinished
            } else if(otherForkIssue) {
              return "There was an error with a fork of a test"
            } else {
              return null
            }
          }
          

          Patrick Ruckstuhl added a comment - Here's my workaround for scriped pipeline. It requires some script security exemptions but otherwise works quite nicely. I'm also trying to print out which test class had the issue. def detectedErrors = anyErrorDetected() if (detectedErrors){ error detectedErrors } @com.cloudbees.groovy.cps.NonCPS def anyErrorDetected() { logReader = currentBuild.rawBuild.logReader def startedNotFinished = [] def forkTimeout = false def otherForkIssue = false logReader.eachLine { def startMatcher = it =~ /.*Running (?<name>[\w.-]+)/ def endMatcher = it =~ /.*Tests run.* in (?<name>[\w.-]+)/ if (startMatcher.matches()){ startedNotFinished.add(startMatcher.group( "name" )) } if (endMatcher.matches()){ startedNotFinished.remove(endMatcher.group( "name" )) } if (it =~ /There was a timeout or other error in the fork/){ forkTimeout = true } else if (it =~ /Error occurred in starting fork/){ otherForkIssue = true } } if (forkTimeout){ return "There was a timeout or other error in the fork. Potential tests with issue: " + startedNotFinished } else if (otherForkIssue) { return "There was an error with a fork of a test" } else { return null } }

          There is a similar issue if the forked process is killed (for example due to an OOM):

          03:34:21  Killed
          03:34:21  [INFO] 
          03:34:21  [INFO] Results:
          03:34:21  [INFO] 
          03:34:21  [WARNING] Tests run: 1183, Failures: 0, Errors: 0, Skipped: 5
          [...]
          03:34:21  Error occurred in starting fork, check output in log
          03:34:21  Process Exit Code: 137
          03:34:21  Crashed tests:
          

          I helped myself with the text finder plugin:

          findText(textFinders: [
            textFinder(alsoCheckConsoleOutput: true, buildResult: 'UNSTABLE', regexp: 'There was a timeout or other error in the fork'),
            textFinder(alsoCheckConsoleOutput: true, buildResult: 'UNSTABLE', regexp: 'Error occurred in starting fork'),
          ])
          

          Tobias Gruetzmacher added a comment - There is a similar issue if the forked process is killed (for example due to an OOM): 03:34:21 Killed 03:34:21 [INFO] 03:34:21 [INFO] Results: 03:34:21 [INFO] 03:34:21 [WARNING] Tests run: 1183, Failures: 0, Errors: 0, Skipped: 5 [...] 03:34:21 Error occurred in starting fork, check output in log 03:34:21 Process Exit Code: 137 03:34:21 Crashed tests: I helped myself with the text finder plugin : findText(textFinders: [ textFinder(alsoCheckConsoleOutput: true , buildResult: 'UNSTABLE' , regexp: 'There was a timeout or other error in the fork' ), textFinder(alsoCheckConsoleOutput: true , buildResult: 'UNSTABLE' , regexp: 'Error occurred in starting fork' ), ])

            Unassigned Unassigned
            bondolo Mike Duigou
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: