-
Bug
-
Resolution: Unresolved
-
Critical
A mostly unresponsive 1.509.4 instance shows in its thread dump:
"Handling GET /plugin/disk-usage/ : http-8088-1" Id=26 Group=main RUNNABLE at java.lang.String.intern(Native Method) at org.xmlpull.mxp1.MXParserCachingStrings.newStringIntern(MXParserCachingStrings.java:317) at org.xmlpull.mxp1.MXParserCachingStrings.newString(MXParserCachingStrings.java:287) at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1779) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135) at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87) at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:261) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71) at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85) at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61) at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:333) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:275) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061) at hudson.util.XStream2.unmarshal(XStream2.java:109) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045) at hudson.XmlFile.unmarshal(XmlFile.java:166) at hudson.model.Run.reload(Run.java:310) at hudson.model.Run.<init>(Run.java:299) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:182) at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54) at hudson.maven.MavenBuild.<init>(MavenBuild.java:113) at sun.reflect.GeneratedConstructorAccessor2641.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1129) at hudson.model.AbstractProject$1.create(AbstractProject.java:325) at hudson.model.AbstractProject$1.create(AbstractProject.java:323) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529) at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41) at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264) at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source) at java.util.AbstractMap$2$1.hasNext(Unknown Source) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165) "Handling GET /plugin/disk-usage/ : http-8088-14" Id=20039 Group=main BLOCKED on hudson.model.RunMap@4e1b74e0 owned by "Handling GET /plugin/disk-usage/ : http-8088-1" Id=26 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - blocked on hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529) at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41) at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264) at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source) at java.util.AbstractMap$2$1.hasNext(Unknown Source) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165)
(and some more threads waiting for the same lock).
Iterating all builds of a project, especially from an HTTP handling thread, is unacceptable. If this information can be reasonably cached, then great; or ProgressiveRendering could be used to display disk usage for a particular project when the user is really interested (with the calculation being aborted if and when the user navigates away); otherwise the functionality must be removed entirely.
[JENKINS-21003] ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems breaks lazy-loading
Ryan Campbell
added a comment - Same problem here. CPU was at 200% thanks to about 27 threads trying to calculate disk usage for the same project. Disabling the disk-usage plugin fixed the issue.
I think this is critical to fix because it's problems like this that give Jenkins a bad name.
Ryan Campbell
added a comment - Same problem here. CPU was at 200% thanks to about 27 threads trying to calculate disk usage for the same project. Disabling the disk-usage plugin fixed the issue.
I think this is critical to fix because it's problems like this that give Jenkins a bad name. 
Lucie Votypkova
added a comment - Hi,
I can cashed information about build size, it is not problem.
there should be max 3 threads for calculation disk usage (calculation of workspaces, calculation of jobs, calculation of builds). I agree this is a serious bug, I will write tests for it so it will not happen again.
Lucie Votypkova
added a comment - Hi,
I can cashed information about build size, it is not problem.
there should be max 3 threads for calculation disk usage (calculation of workspaces, calculation of jobs, calculation of builds). I agree this is a serious bug, I will write tests for it so it will not happen again. Lucie Votypkova
added a comment - Oh sorry I read badly your comment recampbell. I checked the code and cashed information about build size is not problem, but the problem will be a filtering. Pluigin provides a filter build disk usage by age of build - this is necessary feature for our instance to identify jobs which old builds take a lot of disk space.
Lucie Votypkova
added a comment - Oh sorry I read badly your comment recampbell. I checked the code and cashed information about build size is not problem, but the problem will be a filtering. Pluigin provides a filter build disk usage by age of build - this is necessary feature for our instance to identify jobs which old builds take a lot of disk space. Another thing is recalculation threads - they break lazy loading too, it is possible to disable then, but I think that I should do something with them too.
Lucie Votypkova
added a comment - - edited Another thing is recalculation threads - they break lazy loading too, it is possible to disable then, but I think that I should do something with them too. Lucie Votypkova
added a comment - Moving information into separate xml file diskUsage.xml from build.xml should solve problems with lazy loading. During starting all information will be cashed from diskUsage.xml files of non loaded builds. Recalculation thread will update only diskUsage.xml job and update cashed information from nonloaded builds. I hope tested implementation will be soon.
Lucie Votypkova
added a comment - Moving information into separate xml file diskUsage.xml from build.xml should solve problems with lazy loading. During starting all information will be cashed from diskUsage.xml files of non loaded builds. Recalculation thread will update only diskUsage.xml job and update cashed information from nonloaded builds. I hope tested implementation will be soon. Why not keep a single file per job listing disk usage for every build? (Possibly in properties format, if the content is sufficiently simple.) This would be a lot faster to load than hundreds of little XML files. Of course you would need to update it when a new build is created; unnecessary to do so when an old build is deleted, since you can just ignore entries for builds whose directories (Run.id) do not still exist.
Lucie Votypkova
added a comment - It is easiest solution for backward compatibility (it does not need a lot of changes so there is not a big chance for mistake). It is true that your suggestion is faster for loading, I will check the code how complicated it will be. Thanks!
Lucie Votypkova
added a comment - It is easiest solution for backward compatibility (it does not need a lot of changes so there is not a big chance for mistake). It is true that your suggestion is faster for loading, I will check the code how complicated it will be. Thanks! I created PR 28 which prevents duplicate calls to ProjectDiskUsageAction.getJobRootDirDiskUsage which is an expensive operation. This will not prevent lazy loading operation, however the jobs root disk usage will not be calculated for each build.
Steven Christou
added a comment - - edited I created PR 28 which prevents duplicate calls to ProjectDiskUsageAction.getJobRootDirDiskUsage which is an expensive operation. This will not prevent lazy loading operation, however the jobs root disk usage will not be calculated for each build. Lucie Votypkova
added a comment - Now Disk usage should be able to work without loading builds all the time in the new version 0.26 (will be released this month). Only one exception is the first recalcualtion (which is done by recalculation threads). Once it is done disk usage should not need loading build for its work. Please share if this problem disappears. Thanks
Lucie Votypkova
added a comment - Now Disk usage should be able to work without loading builds all the time in the new version 0.26 (will be released this month). Only one exception is the first recalcualtion (which is done by recalculation threads). Once it is done disk usage should not need loading build for its work. Please share if this problem disappears. Thanks
Similar problem here with Jenkins LTS 1.532.1 and Disk Usage 0.23