Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56374

JUnitResultArchiver parsing XML in sort()->compare(), killing Jenkins

      Jenkins 2.150.1

      Jenkins dies with the JVM going into perpetual ergonomics GC. The JVM has a 4Gb heap. The biggest XML file is 180Mb. So that's not a fundamental limitation. Something dumb is going on.

      I think it might be repeatedly parsing the same XML file again and again, because when I dump jstack, I've seen a lot of different XML parsers there. This MIGHT be triggered by a @Parameterized junit test with 50,000 cases. In any case, this has made Jenkins totally unusable.

      It also dies when we have a single JUnit case with a 1Gb output XML, but we have worked around that. But we cannot reduce the size of our test suite just to keep Jenkins happy.

      Reported critical as this has stopped us from using Jenkins. Tempted to report blocker.

      Here is some stack.

      ```
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:180)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
      at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
      at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
      at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:475)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:327)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      at hudson.util.XStream2.unmarshal(XStream2.java:161)
      at hudson.util.XStream2.unmarshal(XStream2.java:132)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
      at hudson.XmlFile.read(XmlFile.java:147)
      at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208)
      at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)

      • locked <0x0000000764f440a8> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
        ...
        ...
        ...
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006ef085628> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006e46233c0> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006d144c000> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
      • locked <0x00000006c3ce5168> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406)
        at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434)
        at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754)
        at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752)
        at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
        at java.util.TimSort.sort(TimSort.java:234)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1462)
        at java.util.Collections.sort(Collections.java:175)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:780)
        at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108)
      • locked <0x00000006d144c120> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85)
        at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175)
      • locked <0x00000006c3d13a18> (a hudson.model.FreeStyleBuild)
        at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
        ```

          [JENKINS-56374] JUnitResultArchiver parsing XML in sort()->compare(), killing Jenkins

          Shevek . added a comment -

          Related stack:

          ```

          Mar 03, 2019 10:00:07 PM hudson.tasks.junit.TestResultAction load
          WARNING: Failed to load /home/jenkins/jobs/XXX-pr/builds/6739/junitResult.xml
          java.io.IOException: Unable to read /home/jenkins/jobs/XXX-pr/builds/6739/junitResult.xml
          at hudson.XmlFile.read(XmlFile.java:149)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)

          ...

          ...

          ...

          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
          at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
          at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
          at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
          at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
          at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
          at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
          at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
          at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406)
          at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434)
          at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754)
          at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752)
          at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
          at java.util.TimSort.sort(TimSort.java:234)
          at java.util.Arrays.sort(Arrays.java:1512)
          at java.util.ArrayList.sort(ArrayList.java:1462)
          at java.util.Collections.sort(Collections.java:175)
          at hudson.tasks.junit.TestResult.freeze(TestResult.java:780)
          at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108)
          at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85)
          at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175)
          at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154)
          at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
          at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
          at hudson.model.Build$BuildExecution.post2(Build.java:186)
          at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
          at hudson.model.Run.execute(Run.java:1835)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
          Caused by: java.lang.OutOfMemoryError: Java heap space

          ```

          Shevek . added a comment - Related stack: ``` Mar 03, 2019 10:00:07 PM hudson.tasks.junit.TestResultAction load WARNING: Failed to load /home/jenkins/jobs/XXX-pr/builds/6739/junitResult.xml java.io.IOException: Unable to read /home/jenkins/jobs/XXX-pr/builds/6739/junitResult.xml at hudson.XmlFile.read(XmlFile.java:149) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) ... ... ... at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641) at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462) at hudson.tasks.junit.TestResult.freeze(TestResult.java:748) at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145) at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63) at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252) at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146) at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416) at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479) at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406) at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434) at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754) at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) at java.util.TimSort.sort(TimSort.java:234) at java.util.Arrays.sort(Arrays.java:1512) at java.util.ArrayList.sort(ArrayList.java:1462) at java.util.Collections.sort(Collections.java:175) at hudson.tasks.junit.TestResult.freeze(TestResult.java:780) at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108) at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85) at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175) at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) at hudson.model.Build$BuildExecution.post2(Build.java:186) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) at hudson.model.Run.execute(Run.java:1835) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: java.lang.OutOfMemoryError: Java heap space ```

          Shevek . added a comment -

          Note that this is the ONLY executor running, we don't run anything else in-process, and the XML file it's failing to load is only 100Mb.

           

          Wait... are we parsing XML in the compare() routine of a sort-object? What the $BADWORD...? WAT?

          Shevek . added a comment - Note that this is the ONLY executor running, we don't run anything else in-process, and the XML file it's failing to load is only 100Mb.   Wait... are we parsing XML in the compare() routine of a sort-object? What the $BADWORD...? WAT?

          Shevek . added a comment -

          jmap -histo:live output:

           

          num #instances #bytes class name
          ----------------------------------------------
          1: 2678703 2691823496 [C
          2: 2678264 64278336 java.lang.String
          3: 811096 58398912 hudson.tasks.junit.CaseResult
          4: 319187 10213984 java.util.HashMap$Node
          5: 192771 8994440 [Ljava.lang.String;
          6: 42153 7186960 [Ljava.lang.Object;
          7: 187092 4490208 com.thoughtworks.xstream.io.path.Path
          8: 12333 2778840 [Ljava.util.HashMap$Node;
          9: 18705 2075528 java.lang.Class
          10: 61023 1952736 java.util.concurrent.ConcurrentHashMap$Node

           

          Where did we get 811,096 CaseResults in the heap from? They're all live objects, too. This looks like a serious memory leak.

          Shevek . added a comment - jmap -histo:live output:   num #instances #bytes class name ---------------------------------------------- 1: 2678703 2691823496 [C 2: 2678264 64278336 java.lang.String 3: 811096 58398912 hudson.tasks.junit.CaseResult 4: 319187 10213984 java.util.HashMap$Node 5: 192771 8994440 [Ljava.lang.String; 6: 42153 7186960 [Ljava.lang.Object; 7: 187092 4490208 com.thoughtworks.xstream.io.path.Path 8: 12333 2778840 [Ljava.util.HashMap$Node; 9: 18705 2075528 java.lang.Class 10: 61023 1952736 java.util.concurrent.ConcurrentHashMap$Node   Where did we get 811,096 CaseResults in the heap from? They're all live objects, too. This looks like a serious memory leak.

          Shevek . added a comment -

          After the GC-storm, we get this:

          num #instances #bytes class name
          ----------------------------------------------
          1: 208769 27421360 [C
          2: 208315 4999560 java.lang.String
          3: 81032 2593024 java.util.HashMap$Node
          4: 37398 2486496 [Ljava.lang.Object;
          5: 9331 2316560 [B
          6: 18857 2092720 java.lang.Class
          7: 62775 2008800 java.util.concurrent.ConcurrentHashMap$Node
          8: 20596 1812448 java.lang.reflect.Method
          9: 12253 1242400 [Ljava.util.HashMap$Node;
          10: 11491 947256 [Ljava.util.WeakHashMap$Entry;

           

          which is a lot saner, and shows that all the data held in memory was related to the memory inefficiency/leak in the reporter, and is not part of normal operation.

          Shevek . added a comment - After the GC-storm, we get this: num #instances #bytes class name ---------------------------------------------- 1: 208769 27421360 [C 2: 208315 4999560 java.lang.String 3: 81032 2593024 java.util.HashMap$Node 4: 37398 2486496 [Ljava.lang.Object; 5: 9331 2316560 [B 6: 18857 2092720 java.lang.Class 7: 62775 2008800 java.util.concurrent.ConcurrentHashMap$Node 8: 20596 1812448 java.lang.reflect.Method 9: 12253 1242400 [Ljava.util.HashMap$Node; 10: 11491 947256 [Ljava.util.WeakHashMap$Entry;   which is a lot saner, and shows that all the data held in memory was related to the memory inefficiency/leak in the reporter, and is not part of normal operation.

          Shevek . added a comment -

          Attached visualvm dump of retained-object sizes showing the suites in memory.

          Shevek . added a comment - Attached visualvm dump of retained-object sizes showing the suites in memory.

          I have the same issue with the latest LTE release. My Jetty is forced to restart frequently because of out of heap memory.

          Torsten Tuchscheerer added a comment - I have the same issue with the latest LTE release. My Jetty is forced to restart frequently because of out of heap memory.

            menonvarun Varun Menon
            shevek Shevek .
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: