Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64479

deadlock in logging and classloading

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      the simple Statement

      Object o = new MyCustomObject();
      Logger.log(Level.FINE, "hello from {0}" o);
      

      can perform classloading depengin on the implementation of MyCustomObject.toString()

      ClassLoading itself should not cause any logging to be output - as if it does this can easilly cause deadlocks (as the classload will have some locks, and logging if going to a Stream will also lock and....)

      However the bytecode-compatability-transformer does use logging (if enabled at FINE level or below.)

      This can cause some deadlocks when you least expect it (but mostly at statup)

      e.g.

      priority:5 - threadId:0x00007f50f0001000 - nativeId:0x96fa - nativeId (decimal):38650 - state:BLOCKED
      stackTrace:
      java.lang.Thread.State: BLOCKED (on object monitor)
      at jenkins.ClassLoaderReflectionToolkit._findLoadedClass(ClassLoaderReflectionToolkit.java:70)
      - waiting to lock <0x00000006cc50fb28> (a hudson.ClassicPluginStrategy$AntClassLoader2)
      at hudson.ClassicPluginStrategy$DependencyClassLoader.findClass(ClassicPluginStrategy.java:632)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      - locked <0x00000006cc65b0d8> (a hudson.ClassicPluginStrategy$DependencyClassLoader)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1406)
      at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1085)
      - locked <0x00000006cc65b068> (a hudson.ClassicPluginStrategy$AntClassLoader2)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      at com.cloudbees.opscenter.server.model.ManagedMaster$State.<clinit>(ManagedMaster.java:96)
      at com.cloudbees.opscenter.server.model.ManagedMaster.newPersistedState(ManagedMaster.java:650)
      at com.cloudbees.opscenter.server.model.ManagedMaster.newPersistedState(ManagedMaster.java:82)
      at com.cloudbees.opscenter.server.model.ConnectedMaster.getPersistedState(ConnectedMaster.java:407)
      at com.cloudbees.opscenter.server.model.ManagedMaster.getPersistedState(ManagedMaster.java:658)
      at com.cloudbees.opscenter.server.model.ManagedMaster.getPersistedState(ManagedMaster.java:82)
      at com.cloudbees.opscenter.server.model.ConnectedMaster.toString(ConnectedMaster.java:346)
      at java.text.MessageFormat.subformat(MessageFormat.java:1280)
      at java.text.MessageFormat.format(MessageFormat.java:865)
      at java.text.Format.format(Format.java:157)
      at java.text.MessageFormat.format(MessageFormat.java:841)
      at java.util.logging.Formatter.formatMessage(Formatter.java:138)
      - locked <0x00000007a0ba6270> (a io.jenkins.lib.support_log_formatter.SupportLogFormatter)
      at io.jenkins.lib.support_log_formatter.SupportLogFormatter.format(SupportLogFormatter.java:90)
      at java.util.logging.StreamHandler.publish(StreamHandler.java:211)
      - locked <0x00000007a0ba6238> (a java.util.logging.StreamHandler)
      at com.cloudbees.jenkins.support.impl.CustomLogs$LogFile.publish(CustomLogs.java:117)
      at com.cloudbees.jenkins.support.impl.CustomLogs$CustomHandler.publish(CustomLogs.java:158)
      at java.util.logging.Logger.log(Logger.java:738)
      at java.util.logging.Logger.doLog(Logger.java:765)
      at java.util.logging.Logger.log(Logger.java:831)
      at com.cloudbees.jce.masterprovisioning.ClusterEndpointListener$ConnectedMasterEndpointIndexerImpl.onModified(ClusterEndpointListener.java:71)
      at com.cloudbees.opscenter.server.model.ConnectedMasterListener$5.f(ConnectedMasterListener.java:124)
      at com.cloudbees.opscenter.server.model.ConnectedMasterListener.fire(ConnectedMasterListener.java:136)
      at com.cloudbees.opscenter.server.model.ConnectedMasterListener.fireOnModified(ConnectedMasterListener.java:122)
      at com.cloudbees.opscenter.server.model.ConnectedMaster.onModified(ConnectedMaster.java:454)
      at com.cloudbees.opscenter.server.model.ConnectedMaster$PropertyList.setOwner(ConnectedMaster.java:2311)
      at com.cloudbees.opscenter.server.model.ConnectedMaster.init(ConnectedMaster.java:513)
      at com.cloudbees.opscenter.server.model.ConnectedMaster.onLoad(ConnectedMaster.java:426)
      at com.cloudbees.opscenter.server.model.ManagedMaster.onLoad(ManagedMaster.java:683)
      at hudson.model.Items.load(Items.java:380)
      at jenkins.model.Jenkins$14.run(Jenkins.java:3279)
      at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      
      OperationsCenter.afterExtensionsAugmented
      priority:5 - threadId:0x00007f514000c000 - nativeId:0x96ec - nativeId (decimal):38636 - state:BLOCKED
      stackTrace:
      java.lang.Thread.State: BLOCKED (on object monitor)
      at java.util.logging.StreamHandler.publish(StreamHandler.java:206)
      - waiting to lock <0x00000007a0ba6238> (a java.util.logging.StreamHandler)
      at com.cloudbees.jenkins.support.impl.CustomLogs$LogFile.publish(CustomLogs.java:117)
      at com.cloudbees.jenkins.support.impl.CustomLogs$CustomHandler.publish(CustomLogs.java:158)
      at java.util.logging.Logger.log(Logger.java:738)
      at java.util.logging.Logger.doLog(Logger.java:765)
      at java.util.logging.Logger.log(Logger.java:851)
      at org.jenkinsci.bytecode.Transformer.transform(Transformer.java:79)
      at hudson.ClassicPluginStrategy$AntClassLoader2.defineClassFromData(ClassicPluginStrategy.java:714)
      at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1326)
      at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1377)
      at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1342)
      at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1089)
      - locked <0x00000006cc50fb28> (a hudson.ClassicPluginStrategy$AntClassLoader2)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      at org.mapdb.SerializerBase.assertSerializable(SerializerBase.java:1480)
      at org.mapdb.BTreeMap.<init>(BTreeMap.java:501)
      at org.mapdb.BTreeMap.preinitCatalog(BTreeMap.java:178)
      at org.mapdb.DB.reinit(DB.java:88)
      at org.mapdb.DB.<init>(DB.java:83)
      at org.mapdb.DBMaker.make(DBMaker.java:704)
      at com.cloudbees.opscenter.utils.MapDbHelper.doCreateDb(MapDbHelper.java:112)
      at com.cloudbees.opscenter.utils.MapDbHelper.create(MapDbHelper.java:54)
      at com.cloudbees.opscenter.server.messaging.Transport.openStore(Transport.java:130)
      at com.cloudbees.opscenter.server.messaging.Transport.open(Transport.java:104)
      - locked <0x00000007765dd0e8> (a com.cloudbees.opscenter.server.messaging.Transport)
      at com.cloudbees.opscenter.server.messaging.Transport.start(Transport.java:726)
      at com.cloudbees.opscenter.server.model.OperationsCenter.afterExtensionsAugmented(OperationsCenter.java:1135)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      at jenkins.model.Jenkins$5.runTask(Jenkins.java:1129)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Issue Links

            Activity

            teilo James Nord created issue -
            teilo James Nord made changes -
            Field Original Value New Value
            Assignee Emilio Escobar [ escoem ] James Nord [ teilo ]
            teilo James Nord made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            teilo James Nord made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            teilo James Nord made changes -
            Remote Link This issue links to "PR#17 (Web Link)" [ 26374 ]
            teilo James Nord made changes -
            Link This issue relates to INFRA-2858 [ INFRA-2858 ]
            jglick Jesse Glick made changes -
            Labels classloader deadlock
            teilo James Nord made changes -
            Remote Link This issue links to "PR #19 (BCT) preferred alternative (Web Link)" [ 26501 ]
            teilo James Nord made changes -
            Remote Link This issue links to "Jetty issue #5950 (Web Link)" [ 26502 ]
            teilo James Nord made changes -
            Link This issue relates to JENKINS-23784 [ JENKINS-23784 ]

              People

              Assignee:
              teilo James Nord
              Reporter:
              teilo James Nord
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: