Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56374

JUnitResultArchiver parsing XML in sort()->compare(), killing Jenkins

      Jenkins 2.150.1

      Jenkins dies with the JVM going into perpetual ergonomics GC. The JVM has a 4Gb heap. The biggest XML file is 180Mb. So that's not a fundamental limitation. Something dumb is going on.

      I think it might be repeatedly parsing the same XML file again and again, because when I dump jstack, I've seen a lot of different XML parsers there. This MIGHT be triggered by a @Parameterized junit test with 50,000 cases. In any case, this has made Jenkins totally unusable.

      It also dies when we have a single JUnit case with a 1Gb output XML, but we have worked around that. But we cannot reduce the size of our test suite just to keep Jenkins happy.

      Reported critical as this has stopped us from using Jenkins. Tempted to report blocker.

      Here is some stack.

      ```
      at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:180)
      at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
      at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
      at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
      at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:475)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:327)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
      at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
      at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
      at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
      at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
      at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
      at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
      at hudson.util.XStream2.unmarshal(XStream2.java:161)
      at hudson.util.XStream2.unmarshal(XStream2.java:132)
      at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
      at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
      at hudson.XmlFile.read(XmlFile.java:147)
      at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208)
      at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)

      • locked <0x0000000764f440a8> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
        ...
        ...
        ...
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006ef085628> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006e46233c0> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
      • locked <0x00000006d144c000> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
        at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
        at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
      • locked <0x00000006c3ce5168> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
        at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
        at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
        at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
        at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
        at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406)
        at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434)
        at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754)
        at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752)
        at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
        at java.util.TimSort.sort(TimSort.java:234)
        at java.util.Arrays.sort(Arrays.java:1512)
        at java.util.ArrayList.sort(ArrayList.java:1462)
        at java.util.Collections.sort(Collections.java:175)
        at hudson.tasks.junit.TestResult.freeze(TestResult.java:780)
        at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108)
      • locked <0x00000006d144c120> (a hudson.tasks.junit.TestResultAction)
        at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85)
        at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175)
      • locked <0x00000006c3d13a18> (a hudson.model.FreeStyleBuild)
        at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1835)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
        ```

          [JENKINS-56374] JUnitResultArchiver parsing XML in sort()->compare(), killing Jenkins

          Shevek . created issue -
          Shevek . made changes -
          Environment Original: Jenkins 2.150.1

          Jenkins dies with the JVM going into perpetual ergonomics GC. The JVM has a 4Gb heap. The biggest XML file is 180Mb. So that's not a fundamental limitation. Something dumb is going on.

          I _think_ it might be repeatedly parsing the same XML file again and again, because when I dump jstack, I've seen a lot of different XML parsers there. This MIGHT be triggered by a @Parameterized junit test with 50,000 cases. In any case, this has made Jenkins totally unusable.

          It also dies when we have a single JUnit case with a 1Gb output XML, but we have worked around that. But we cannot reduce the size of our test suite just to keep Jenkins happy.

          Reported critical as this has stopped us from using Jenkins. Tempted to report blocker.

          Here is some stack.

          ```
                  at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:180)
                  at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
                  at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
                  at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
                  at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:475)
                  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:327)
                  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
                  at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
                  at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
                  at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
                  at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
                  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
                  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
                  at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
                  at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
                  at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
                  at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
                  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
                  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
                  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
                  at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
                  at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
                  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
                  at hudson.util.XStream2.unmarshal(XStream2.java:161)
                  at hudson.util.XStream2.unmarshal(XStream2.java:132)
                  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
                  at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
                  at hudson.XmlFile.read(XmlFile.java:147)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
                  - locked <0x0000000764f440a8> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
                  at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
                  at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
                  at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
                  at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
                  at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
                  at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
          ...
          ...
          ...
                  at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
                  - locked <0x00000006ef085628> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
                  at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
                  at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
                  at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
                  at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
                  at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
                  at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
                  - locked <0x00000006e46233c0> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
                  at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
                  at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
                  at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
                  at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
                  at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
                  at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
                  - locked <0x00000006d144c000> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
                  at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
                  at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
                  at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
                  at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
                  at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
                  at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
                  at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
                  - locked <0x00000006c3ce5168> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
                  at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
                  at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
                  at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
                  at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
                  at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406)
                  at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434)
                  at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754)
                  at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752)
                  at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
                  at java.util.TimSort.sort(TimSort.java:234)
                  at java.util.Arrays.sort(Arrays.java:1512)
                  at java.util.ArrayList.sort(ArrayList.java:1462)
                  at java.util.Collections.sort(Collections.java:175)
                  at hudson.tasks.junit.TestResult.freeze(TestResult.java:780)
                  at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108)
                  - locked <0x00000006d144c120> (a hudson.tasks.junit.TestResultAction)
                  at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85)
                  at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175)
                  - locked <0x00000006c3d13a18> (a hudson.model.FreeStyleBuild)
                  at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154)
                  at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                  at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                  at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                  at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                  at hudson.model.Build$BuildExecution.post2(Build.java:186)
                  at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                  at hudson.model.Run.execute(Run.java:1835)
                  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                  at hudson.model.ResourceController.execute(ResourceController.java:97)
                  at hudson.model.Executor.run(Executor.java:429)
          ```
          Shevek . made changes -
          Description New: Jenkins 2.150.1

          Jenkins dies with the JVM going into perpetual ergonomics GC. The JVM has a 4Gb heap. The biggest XML file is 180Mb. So that's not a fundamental limitation. Something dumb is going on.

          I _think_ it might be repeatedly parsing the same XML file again and again, because when I dump jstack, I've seen a lot of different XML parsers there. This MIGHT be triggered by a @Parameterized junit test with 50,000 cases. In any case, this has made Jenkins totally unusable.

          It also dies when we have a single JUnit case with a 1Gb output XML, but we have worked around that. But we cannot reduce the size of our test suite just to keep Jenkins happy.

          Reported critical as this has stopped us from using Jenkins. Tempted to report blocker.

          Here is some stack.

          ```
           at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:180)
           at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
           at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
           at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
           at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:475)
           at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:327)
           at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
           at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
           at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
           at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
           at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
           at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
           at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
           at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
           at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
           at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
           at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
           at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
           at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
           at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
           at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
           at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
           at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
           at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
           at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
           at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
           at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
           at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
           at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
           at hudson.util.XStream2.unmarshal(XStream2.java:161)
           at hudson.util.XStream2.unmarshal(XStream2.java:132)
           at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
           at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
           at hudson.XmlFile.read(XmlFile.java:147)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:208)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
           - locked <0x0000000764f440a8> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
           at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
           at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
           at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
           at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
           at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
           at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
          ...
          ...
          ...
           at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
           - locked <0x00000006ef085628> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
           at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
           at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
           at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
           at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
           at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
           at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
           - locked <0x00000006e46233c0> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
           at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
           at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
           at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
           at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
           at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
           at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:138)
           - locked <0x00000006d144c000> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
           at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
           at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
           at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
           at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
           at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:641)
           at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:462)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:748)
           at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:213)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:145)
           - locked <0x00000006c3ce5168> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:63)
           at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:252)
           at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:146)
           at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:416)
           at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:479)
           at hudson.tasks.junit.CaseResult.getFailedSince(CaseResult.java:406)
           at hudson.tasks.junit.CaseResult.getAge(CaseResult.java:434)
           at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:754)
           at hudson.tasks.junit.CaseResult$1.compare(CaseResult.java:752)
           at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
           at java.util.TimSort.sort(TimSort.java:234)
           at java.util.Arrays.sort(Arrays.java:1512)
           at java.util.ArrayList.sort(ArrayList.java:1462)
           at java.util.Collections.sort(Collections.java:175)
           at hudson.tasks.junit.TestResult.freeze(TestResult.java:780)
           at hudson.tasks.junit.TestResultAction.setResult(TestResultAction.java:108)
           - locked <0x00000006d144c120> (a hudson.tasks.junit.TestResultAction)
           at hudson.tasks.junit.TestResultAction.<init>(TestResultAction.java:85)
           at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:175)
           - locked <0x00000006c3d13a18> (a hudson.model.FreeStyleBuild)
           at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:154)
           at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
           at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
           at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
           at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
           at hudson.model.Build$BuildExecution.post2(Build.java:186)
           at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
           at hudson.model.Run.execute(Run.java:1835)
           at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
           at hudson.model.ResourceController.execute(ResourceController.java:97)
           at hudson.model.Executor.run(Executor.java:429)
          ```
          Shevek . made changes -
          Summary Original: JUnitResultArchiver crazy resource inefficient, killing Jenkins New: JUnitResultArchiver parsing XML in sort()->compare(), killing Jenkins
          Shevek . made changes -
          Labels New: memory-leak
          Shevek . made changes -
          Environment New: Linux
          Java 8
          Jenkins 2.150.1
          JUnit plugin 1.27
          xUnit plugin 2.3.2
          Shevek . made changes -
          Attachment New: jenkins-memory.png [ 46246 ]

            menonvarun Varun Menon
            shevek Shevek .
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: