Details
-
Type:
Bug
-
Status: Open (View Workflow)
-
Priority:
Critical
-
Resolution: Unresolved
-
Component/s: disk-usage-plugin
-
Labels:
-
Similar Issues:
Description
A mostly unresponsive 1.509.4 instance shows in its thread dump:
"Handling GET /plugin/disk-usage/ : http-8088-1" Id=26 Group=main RUNNABLE at java.lang.String.intern(Native Method) at org.xmlpull.mxp1.MXParserCachingStrings.newStringIntern(MXParserCachingStrings.java:317) at org.xmlpull.mxp1.MXParserCachingStrings.newString(MXParserCachingStrings.java:287) at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1779) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148) at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135) at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87) at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:261) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71) at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85) at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61) at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:333) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:275) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:222) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1061) at hudson.util.XStream2.unmarshal(XStream2.java:109) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1045) at hudson.XmlFile.unmarshal(XmlFile.java:166) at hudson.model.Run.reload(Run.java:310) at hudson.model.Run.<init>(Run.java:299) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:182) at hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54) at hudson.maven.MavenBuild.<init>(MavenBuild.java:113) at sun.reflect.GeneratedConstructorAccessor2641.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1129) at hudson.model.AbstractProject$1.create(AbstractProject.java:325) at hudson.model.AbstractProject$1.create(AbstractProject.java:323) at hudson.model.RunMap.retrieve(RunMap.java:225) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:636) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368) at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:220) at hudson.tasks.Fingerprinter$FingerprintAction.compact(Fingerprinter.java:356) at hudson.tasks.Fingerprinter$FingerprintAction.onLoad(Fingerprinter.java:345) at hudson.model.Run.onLoad(Run.java:323) at hudson.model.RunMap.retrieve(RunMap.java:226) at hudson.model.RunMap.retrieve(RunMap.java:59) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - locked hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529) at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41) at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264) at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source) at java.util.AbstractMap$2$1.hasNext(Unknown Source) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165) "Handling GET /plugin/disk-usage/ : http-8088-14" Id=20039 Group=main BLOCKED on hudson.model.RunMap@4e1b74e0 owned by "Handling GET /plugin/disk-usage/ : http-8088-1" Id=26 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:674) - blocked on hudson.model.RunMap@4e1b74e0 at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:657) at jenkins.model.lazy.AbstractLazyLoadRunMap.getById(AbstractLazyLoadRunMap.java:529) at jenkins.model.lazy.BuildReferenceMapAdapter.unwrap(BuildReferenceMapAdapter.java:41) at jenkins.model.lazy.BuildReferenceMapAdapter.access$200(BuildReferenceMapAdapter.java:26) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter._unwrap(BuildReferenceMapAdapter.java:351) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter.access$400(BuildReferenceMapAdapter.java:243) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:266) at jenkins.model.lazy.BuildReferenceMapAdapter$SetAdapter$1.adapt(BuildReferenceMapAdapter.java:264) at hudson.util.AdaptedIterator.next(AdaptedIterator.java:54) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:648) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at java.util.Collections$UnmodifiableCollection$1.hasNext(Unknown Source) at java.util.AbstractMap$2$1.hasNext(Unknown Source) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsageAllSubItems(ProjectDiskUsageAction.java:115) at hudson.plugins.disk_usage.ProjectDiskUsageAction.getBuildsDiskUsage(ProjectDiskUsageAction.java:165)
(and some more threads waiting for the same lock).
Iterating all builds of a project, especially from an HTTP handling thread, is unacceptable. If this information can be reasonably cached, then great; or ProgressiveRendering could be used to display disk usage for a particular project when the user is really interested (with the calculation being aborted if and when the user navigates away); otherwise the functionality must be removed entirely.
Attachments
Activity
Why not keep a single file per job listing disk usage for every build? (Possibly in properties format, if the content is sufficiently simple.) This would be a lot faster to load than hundreds of little XML files. Of course you would need to update it when a new build is created; unnecessary to do so when an old build is deleted, since you can just ignore entries for builds whose directories (Run.id) do not still exist.
It is easiest solution for backward compatibility (it does not need a lot of changes so there is not a big chance for mistake). It is true that your suggestion is faster for loading, I will check the code how complicated it will be. Thanks!
I created PR 28 which prevents duplicate calls to ProjectDiskUsageAction.getJobRootDirDiskUsage which is an expensive operation. This will not prevent lazy loading operation, however the jobs root disk usage will not be calculated for each build.
Now Disk usage should be able to work without loading builds all the time in the new version 0.26 (will be released this month). Only one exception is the first recalcualtion (which is done by recalculation threads). Once it is done disk usage should not need loading build for its work. Please share if this problem disappears. Thanks
Moving information into separate xml file diskUsage.xml from build.xml should solve problems with lazy loading. During starting all information will be cashed from diskUsage.xml files of non loaded builds. Recalculation thread will update only diskUsage.xml job and update cashed information from nonloaded builds. I hope tested implementation will be soon.