Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23514

Deadlock in Xstream at Jenkins startup

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Duplicate
    • core
    • None
    • Jenkins ver. 1.554.2.2 (Jenkins Enterprise by CloudBees 14.05)
      MacOS 10.9.3
      java version "1.7.0_55"

    Description

      Deadlock in com.thoughtworks.xstream.mapper.AnnotationMapper when Jenkins loads folders (see thread dump fragment, "www", "mobile" and "admin"are Jenkins folders).

      Deadlock occur systematically when the stdout and stderr are routed to the shell console. The workaround is to route stdout and stderr to a file.

      Attached:

      • jenkins-startup.log: jenkins stdout & stderr
      • deadlock.log: threaddump highlighting the dead lock in XStream
      Java stack information for the threads listed above:
===================================================
"Loading job www":
	at java.lang.Class.initAnnotationsIfNecessary(Class.java:3175)
	- waiting to lock <0x00000007e22586b0> (a java.lang.Class for hudson.model.Job)
	at java.lang.Class.getAnnotation(Class.java:3137)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:625)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:610)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:181)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.getLocalConverter(AnnotationMapper.java:153)
	at com.thoughtworks.xstream.mapper.MapperWrapper.getLocalConverter(MapperWrapper.java:99)
	at hudson.util.xstream.MapperDelegate.getLocalConverter(MapperDelegate.java:112)
	at com.thoughtworks.xstream.mapper.MapperWrapper.getLocalConverter(MapperWrapper.java:99)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:348)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:289)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:228)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
	at hudson.util.XStream2.unmarshal(XStream2.java:113)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
	at hudson.XmlFile.read(XmlFile.java:142)
	at hudson.model.Items.load(Items.java:275)
	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:105)
	at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:238)
	at hudson.model.Items.load(Items.java:276)
	at jenkins.model.Jenkins$18.run(Jenkins.java:2588)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
	at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
"Loading job admin":
	at java.lang.Class.initAnnotationsIfNecessary(Class.java:3175)
	- waiting to lock <0x00000007e1dce000> (a java.lang.Class for hudson.model.Run)
	at java.lang.Class.getAnnotation(Class.java:3137)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:625)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:610)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$1.add(AnnotationMapper.java:250)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$1.add(AnnotationMapper.java:245)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.addParametrizedTypes(AnnotationMapper.java:279)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:204)
	- locked <0x00000007e22586b0> (a java.lang.Class for hudson.model.Job)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
	at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
	at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
	at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
	at hudson.util.XStream2.unmarshal(XStream2.java:113)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
	at hudson.XmlFile.read(XmlFile.java:142)
	at hudson.model.Items.load(Items.java:275)
	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:105)
	at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:238)
	at hudson.model.Items.load(Items.java:276)
	at jenkins.model.Jenkins$18.run(Jenkins.java:2588)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
	at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
"Loading job mobile":
	at java.lang.Class.initAnnotationsIfNecessary(Class.java:3175)
	- waiting to lock <0x00000007e22586b0> (a java.lang.Class for hudson.model.Job)
	at java.lang.Class.getAnnotation(Class.java:3137)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:625)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$UnprocessedTypesSet.add(AnnotationMapper.java:610)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$1.add(AnnotationMapper.java:250)
	at com.thoughtworks.xstream.mapper.AnnotationMapper$1.add(AnnotationMapper.java:245)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.addParametrizedTypes(AnnotationMapper.java:279)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.processTypes(AnnotationMapper.java:204)
	- locked <0x00000007e1dce000> (a java.lang.Class for hudson.model.Run)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.processAnnotations(AnnotationMapper.java:182)
	at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:141)
	at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
	at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
	at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
	at hudson.util.XStream2.unmarshal(XStream2.java:113)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
	at hudson.XmlFile.read(XmlFile.java:142)
	at hudson.model.Items.load(Items.java:275)
	at hudson.model.ItemGroupMixIn.loadChildren(ItemGroupMixIn.java:105)
	at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:238)
	at hudson.model.Items.load(Items.java:276)
	at jenkins.model.Jenkins$18.run(Jenkins.java:2588)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
	at jenkins.model.Jenkins$7.runTask(Jenkins.java:899)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-23455 Deadlock during startup

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-19561 Unsafe & inefficient concurrency in XStream

          • Critical - Crashes, loss of data, severe memory leak.
          • Fixed but Unreleased
          links to

          Web Link CloudBees Internal DEV-1578

          Activity

            Ascending order - Click to sort in descending order
            danielbeck Daniel Beck added a comment -

            Another workaround might be to set the system property hudson.model.Hudson.parallelLoad to false.

            danielbeck Daniel Beck added a comment - Another workaround might be to set the system property hudson.model.Hudson.parallelLoad to false.
            danielbeck Daniel Beck added a comment -

            This may even be duplicate of JENKINS-19561, given Jesse's comment there, but I'm not sure.

            danielbeck Daniel Beck added a comment - This may even be duplicate of JENKINS-19561 , given Jesse's comment there, but I'm not sure.
            jglick Jesse Glick added a comment -

            Yeah it is a duplicate—this is one possible symptom.

            jglick Jesse Glick added a comment - Yeah it is a duplicate—this is one possible symptom.

            People

              Unassigned Unassigned
              cleclerc Cyrille Le Clerc
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: