Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15382

Memory exhaustion parsing large test stdio from Surefire

      A user with some *-output.txt files pushing 70Mb received errors when publishing test results:

      [ERROR] FATAL ERROR 
      [INFO] ------------------------------------------------------------------------ 
      [INFO] Java heap space 
      [INFO] ------------------------------------------------------------------------ 
      [INFO] Trace 
      java.lang.OutOfMemoryError: Java heap space 
      	at java.util.Arrays.copyOf(Arrays.java:2882) 
      	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) 
      	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515) 
      	at java.lang.StringBuffer.append(StringBuffer.java:306) 
      	at java.io.StringWriter.write(StringWriter.java:77) 
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129) 
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104) 
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1050) 
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:1075) 
      	at org.apache.commons.io.IOUtils.toString(IOUtils.java:382) 
      	at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1103) 
      	at org.apache.commons.io.FileUtils.readFileToString(FileUtils.java:1120) 
      	at hudson.tasks.junit.SuiteResult.<init>(SuiteResult.java:195) 
      	at hudson.tasks.junit.SuiteResult.parseSuite(SuiteResult.java:130) 
      	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:115) 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:227) 
      	at hudson.tasks.junit.TestResult.parse(TestResult.java:163) 
      	at hudson.maven.reporters.SurefireArchiver.postExecute(SurefireArchiver.java:141) 
      	at hudson.maven.Maven2Builder.postExecute(Maven2Builder.java:155) 
      	at hudson.maven.MavenBuilder$Adapter.postExecute(MavenBuilder.java:310) 
      	at hudson.maven.agent.PluginManagerInterceptor$1MojoIntercepterImpl.callPost(PluginManagerInterceptor.java:170) 
      	at hudson.maven.agent.PluginManagerInterceptor.executeMojo(PluginManagerInterceptor.java:183) 
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556) 
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535) 
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387) 
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348) 
      	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180) 
      	at org.apache.maven.lifecycle.LifecycleExecutorInterceptor.execute(LifecycleExecutorInterceptor.java:65) 
      	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) 
      	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) 
      	at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) 
      [INFO] ------------------------------------------------------------------------ 
      [INFO] ...
      [INFO] Final Memory: 223M/1224M 
      [INFO] ------------------------------------------------------------------------
      

      Indeed it is silly to call FileUtils.readFileToString only to throw away the great majority of what was just read.

          [JENKINS-15382] Memory exhaustion parsing large test stdio from Surefire

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          core/src/main/java/hudson/tasks/junit/SuiteResult.java
          http://jenkins-ci.org/commit/jenkins/4acc879bf56dc727838086c8c93816ad69222552
          Log:
          [FIXED JENKINS-15382] Memory exhaustion parsing large test stdio from Surefire.(cherry picked from commit fe934aac007a20c43e803de61ef8b6cf28c4434f)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/java/hudson/tasks/junit/SuiteResult.java http://jenkins-ci.org/commit/jenkins/4acc879bf56dc727838086c8c93816ad69222552 Log: [FIXED JENKINS-15382] Memory exhaustion parsing large test stdio from Surefire.(cherry picked from commit fe934aac007a20c43e803de61ef8b6cf28c4434f) Conflicts: changelog.html

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          core/src/main/java/hudson/tasks/junit/SuiteResult.java
          core/src/main/java/hudson/tasks/junit/TestResult.java
          core/src/main/java/hudson/util/TextFile.java
          test/src/test/groovy/hudson/util/TextFileTest.groovy
          test/src/test/resources/hudson/util/ascii.txt
          http://jenkins-ci.org/commit/jenkins/14d980c4377dc703df8db76f192de19246a0266e
          Log:
          [FIXED JENKINS-15382]

          Revisiting the fix. Avoid parsing the entire text by only looking at the tail portion.

          Compare: https://github.com/jenkinsci/jenkins/compare/0256f0a74e9b...14d980c4377d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/java/hudson/tasks/junit/SuiteResult.java core/src/main/java/hudson/tasks/junit/TestResult.java core/src/main/java/hudson/util/TextFile.java test/src/test/groovy/hudson/util/TextFileTest.groovy test/src/test/resources/hudson/util/ascii.txt http://jenkins-ci.org/commit/jenkins/14d980c4377dc703df8db76f192de19246a0266e Log: [FIXED JENKINS-15382] Revisiting the fix. Avoid parsing the entire text by only looking at the tail portion. Compare: https://github.com/jenkinsci/jenkins/compare/0256f0a74e9b...14d980c4377d

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2680

          Result = UNSTABLE

          dogfood added a comment - Integrated in jenkins_main_trunk #2680 Result = UNSTABLE

          marked as lts-candidated for last commit http://jenkins-ci.org/commit/jenkins/14d980c4377dc703df8db76f192de19246a0266e to revisit this fix

          Nicolas De Loof added a comment - marked as lts-candidated for last commit http://jenkins-ci.org/commit/jenkins/14d980c4377dc703df8db76f192de19246a0266e to revisit this fix

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/util/AtomicFileWriter.java
          http://jenkins-ci.org/commit/jenkins/f30ee33b5d503e940307974cca4fd09e7a2c50dd
          Log:
          JENKINS-15382 no longer only about UTF-8

          (cherry picked from commit 59f0f32ff63d081475e5c691a8c932d98f688101)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/util/AtomicFileWriter.java http://jenkins-ci.org/commit/jenkins/f30ee33b5d503e940307974cca4fd09e7a2c50dd Log: JENKINS-15382 no longer only about UTF-8 (cherry picked from commit 59f0f32ff63d081475e5c691a8c932d98f688101)

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          core/src/main/java/hudson/tasks/junit/SuiteResult.java
          core/src/main/java/hudson/tasks/junit/TestResult.java
          core/src/main/java/hudson/util/TextFile.java
          test/src/test/groovy/hudson/util/TextFileTest.groovy
          test/src/test/resources/hudson/util/ascii.txt
          http://jenkins-ci.org/commit/jenkins/a3512b9fe983cbbdb3b78597086822a61abb81ff
          Log:
          [FIXED JENKINS-15382]

          Revisiting the fix. Avoid parsing the entire text by only looking at the tail portion.

          (cherry picked from commit 14d980c4377dc703df8db76f192de19246a0266e)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/java/hudson/tasks/junit/SuiteResult.java core/src/main/java/hudson/tasks/junit/TestResult.java core/src/main/java/hudson/util/TextFile.java test/src/test/groovy/hudson/util/TextFileTest.groovy test/src/test/resources/hudson/util/ascii.txt http://jenkins-ci.org/commit/jenkins/a3512b9fe983cbbdb3b78597086822a61abb81ff Log: [FIXED JENKINS-15382] Revisiting the fix. Avoid parsing the entire text by only looking at the tail portion. (cherry picked from commit 14d980c4377dc703df8db76f192de19246a0266e)

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/util/TextFile.java
          http://jenkins-ci.org/commit/jenkins/fca0ea8aa87ab18a1b398ce7bb8ba20bb8bde186
          Log:
          JENKINS-15382 Forgot to close the file

          (cherry picked from commit 43ecda6d2d4aba44144d12fb89ad1ed3e77bc7f3)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/util/TextFile.java http://jenkins-ci.org/commit/jenkins/fca0ea8aa87ab18a1b398ce7bb8ba20bb8bde186 Log: JENKINS-15382 Forgot to close the file (cherry picked from commit 43ecda6d2d4aba44144d12fb89ad1ed3e77bc7f3)

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/tasks/junit/CaseResult.java
          http://jenkins-ci.org/commit/jenkins/bcf9a84f18e42d41f268bd62e6e62d56d2f9b23b
          Log:
          JENKINS-15382 not sure which code is triggering animal sniffer violation

          (cherry picked from commit f7671e621e374bc51a81ec75c056d15ee359093a)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/tasks/junit/CaseResult.java http://jenkins-ci.org/commit/jenkins/bcf9a84f18e42d41f268bd62e6e62d56d2f9b23b Log: JENKINS-15382 not sure which code is triggering animal sniffer violation (cherry picked from commit f7671e621e374bc51a81ec75c056d15ee359093a)

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/util/TextFile.java
          http://jenkins-ci.org/commit/jenkins/36ef7c646734582b18178f83b6bb2c13e598b610
          Log:
          JENKINS-15382 removed Java 1.6 dependency

          (cherry picked from commit 82404407f2c26b3b0509e392ec35dee8ea7dc571)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/util/TextFile.java http://jenkins-ci.org/commit/jenkins/36ef7c646734582b18178f83b6bb2c13e598b610 Log: JENKINS-15382 removed Java 1.6 dependency (cherry picked from commit 82404407f2c26b3b0509e392ec35dee8ea7dc571)

          rajesh r added a comment -

          Still getting the issue while running maven project in jenkins :
          testcase name="testrunners.CukeTestOne" classname="org.apache.maven.surefire.junit4.JUnit4Provider" time="0.767">
          <error message="Java heap space" type="java.lang.OutOfMemoryError">java.lang.OutOfMemoryError: Java heap space
          at java.util.Arrays.copyOf(Arrays.java:2367)
          at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
          at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
          at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
          at java.lang.StringBuffer.append(StringBuffer.java:237)
          at java.io.StringWriter.write(StringWriter.java:112)
          at gherkin.deps.com.google.gson.stream.JsonWriter.string(JsonWriter.java:538)
          at gherkin.deps.com.google.gson.stream.JsonWriter.value(JsonWriter.java:404)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:353)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:337)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96)
          at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209)
          at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146)
          at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68)
          </error>

          rajesh r added a comment - Still getting the issue while running maven project in jenkins : testcase name="testrunners.CukeTestOne" classname="org.apache.maven.surefire.junit4.JUnit4Provider" time="0.767"> <error message="Java heap space" type="java.lang.OutOfMemoryError">java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) at java.lang.StringBuffer.append(StringBuffer.java:237) at java.io.StringWriter.write(StringWriter.java:112) at gherkin.deps.com.google.gson.stream.JsonWriter.string(JsonWriter.java:538) at gherkin.deps.com.google.gson.stream.JsonWriter.value(JsonWriter.java:404) at gherkin.deps.com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:353) at gherkin.deps.com.google.gson.internal.bind.TypeAdapters$13.write(TypeAdapters.java:337) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:96) at gherkin.deps.com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.write(CollectionTypeAdapterFactory.java:60) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:209) at gherkin.deps.com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.write(MapTypeAdapterFactory.java:146) at gherkin.deps.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:68) </error>

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: