A clean start of Hudson takes currently approx 45-65 minutes. The initial loading page "Please wait while Hudson is getting ready to work...." is only visible page during this time. The start time was much lower as before, but dramatically increased after updating the analysis plugins some months ago and has been steadily increasing as we get more builds and more projects in our Hudson installation.

During the startup time, we see the following line, repeating for every build in our system:

---

Sep 2, 2010 4:17:49 PM hudson.plugins.analysis.core.BuildResult loadResult
INFO: Loaded data file /home/hudson/hudson/server_installation/jobs/projectname/builds/2010-08-06_09-31-30/analysis.xml for build 0

---

We have a total of 1114 analysis.xml (for all jobs and builds) totaling 6.7GB.

=> We assume that the analysis-collector plugin in scanning all of these files on every startup.

When we do "kill -3" repeatedly on the Hudson java process during the startup time, we find that the following stack trace is representative of what is going on:
----------------------------------
"Loading job projectname" daemon prio=10 tid=0x0854fc00 nid=0x7bd runnable [0x2f3a0000..0x2f3a2020]
java.lang.Thread.State: RUNNABLE
at com.thoughtworks.xstream.core.util.ClassLoaderReference.loadClass(ClassLoaderReference.java:31)
at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:61)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:71)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:86)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:96)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:138)
at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:38)
at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:56)
at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:70)
at com.thoughtworks.xstream.converters.collections.ArrayConverter.unmarshal(ArrayConverter.java:55)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:137)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:33)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:926)
at hudson.util.XStream2.unmarshal(XStream2.java:79)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:912)
at com.thoughtworks.xstream.XStream.fromXML(XStream.java:856)
at hudson.XmlFile.read(XmlFile.java:126)
at hudson.plugins.analysis.core.BuildResult.loadResult(BuildResult.java:759)
at hudson.plugins.analysis.core.BuildResult.getProject(BuildResult.java:739)

• locked <0xa75e52c0> (a java.lang.Object)
  at hudson.plugins.analysis.core.BuildResult.getContainer(BuildResult.java:883)
  at hudson.plugins.analysis.core.BuildResult.getAnnotations(BuildResult.java:484)
  at hudson.plugins.analysis.collector.AnalysisResult.countAnnotations(AnalysisResult.java:80)
  at hudson.plugins.analysis.collector.AnalysisResult.readResolve(AnalysisResult.java:70)
  at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:46)
  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:181)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
  at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:290)
  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:233)
  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:180)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
  at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
  at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
  at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:61)
  at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
  at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:290)
  at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:233)
  at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:180)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:82)
  at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:63)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:76)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:60)
  at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:137)
  at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:33)
  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:926)
  at hudson.util.XStream2.unmarshal(XStream2.java:79)
  at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:912)
  at hudson.XmlFile.unmarshal(XmlFile.java:148)
  at hudson.model.Run.<init>(Run.java:257)
  at hudson.model.AbstractBuild.<init>(AbstractBuild.java:151)
  at hudson.model.Build.<init>(Build.java:99)
  at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:41)
  at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at hudson.model.AbstractProject.loadBuild(AbstractProject.java:870)
  at hudson.model.AbstractProject$1.create(AbstractProject.java:238)
  at hudson.model.AbstractProject$1.create(AbstractProject.java:236)
  at hudson.model.RunMap.load(RunMap.java:221)
• locked <0x368da870> (a hudson.model.RunMap)
  at hudson.model.AbstractProject.onLoad(AbstractProject.java:236)
  at hudson.model.Project.onLoad(Project.java:87)
  at hudson.model.Items.load(Items.java:106)
  at hudson.model.Hudson$12.run(Hudson.java:2167)
  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:146)
  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:259)
  at hudson.model.Hudson$3.runTask(Hudson.java:679)
  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:187)
  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:94)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:619)
  ----------------------------------

By looking at the code of the analysis collector and other plugin, we believe the problem lies in the method "readResolve" in "plugins/analysis-collector/src/main/java/hudson/plugins/analysis/collector/AnalysisResult.java"