Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27669

Creating an empty logger crashs Jenkins

    XMLWordPrintable

Details

    Description

      With support bundle plugin installed, when I create a new logger and let the logger classname field empty, Jenkins crashes when I click Save button.

      On save, Jenkins starts to write tons of exception logs and is not responsible anymore. Killing the JVM is the only way to stop it doing this.

      Logged exception looks like this one:

      Exception in thread "RequestHandlerThread[#10]" java.lang.ClassCircularityError: hudson/logging/LogRecorder
      	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:507)
      	at java.util.logging.Logger.log(Logger.java:616)
      	at java.util.logging.Logger.doLog(Logger.java:641)
      	at java.util.logging.Logger.log(Logger.java:730)
      	at org.eclipse.jetty.util.log.JavaUtilLog.warn(JavaUtilLog.java:70)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:698)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      On 1.580, faulty configuration is saved on disk. To restart the instance, faulty configuration file $JENKINS_HOME/log/<faulty>.xml must be removed.

      On 1.606, configuration is not saved, so restarting the instance works.

      Attachments

        Issue Links

          Activity

            ydubreuil Yoann Dubreuil created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Assignee Steven Christou [ schristou ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick added a comment -

            Without support-core, it works fine. An empty value is legal, as it is in java.util.logging generally.

            jglick Jesse Glick added a comment - Without support-core , it works fine. An empty value is legal, as it is in java.util.logging generally.
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            jglick Jesse Glick added a comment -

            Yoann can you still reproduce this? I could not. I started a dev version of support-core on 1.606, created a new log recorder, added a logger with the blank name (logging ALL), and saved. It produced lots of messages, to be sure, but Jenkins did not crash.

            jglick Jesse Glick added a comment - Yoann can you still reproduce this? I could not. I started a dev version of support-core on 1.606, created a new log recorder, added a logger with the blank name (logging ALL ), and saved. It produced lots of messages, to be sure, but Jenkins did not crash.
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ] Yoann Dubreuil [ ydubreuil ]

            Jesse, I can't reproduce on a dev environment either (tried on 1.554 core). But I'm still able to reproduce by starting a standalone OSS 1.606 war and installing the latest support-core plugin manually.

            ydubreuil Yoann Dubreuil added a comment - Jesse, I can't reproduce on a dev environment either (tried on 1.554 core). But I'm still able to reproduce by starting a standalone OSS 1.606 war and installing the latest support-core plugin manually.
            jglick Jesse Glick made changes -
            Assignee Yoann Dubreuil [ ydubreuil ] Jesse Glick [ jglick ]
            jglick Jesse Glick added a comment -

            Interesting…after having run 1.606, created the empty logger, deleted it, I tried to install Support Core (2.21) in the same session and got linkage errors, for example

            java.lang.ClassCircularityError: java/util/Map$Entry
            	at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:506)
            	at java.util.logging.Logger.log(Logger.java:738)
            	at java.util.logging.Logger.doLog(Logger.java:765)
            	at java.util.logging.Logger.log(Logger.java:788)
            	at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:108)
            	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:439)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
            	at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:62)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            	at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1397)
            	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1074)
            	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            	at java.lang.Class.getDeclaredMethods0(Native Method)
            	at java.lang.Class.privateGetDeclaredMethods(Class.java:2693)
            	at java.lang.Class.getDeclaredMethods(Class.java:1967)
            	at org.jvnet.hudson.annotation_indexer.Index$2$1.fetch(Index.java:102)
            	at org.jvnet.hudson.annotation_indexer.Index$2$1.hasNext(Index.java:72)
            	at org.jvnet.hudson.annotation_indexer.SubtypeIterator.fetch(SubtypeIterator.java:18)
            	at org.jvnet.hudson.annotation_indexer.SubtypeIterator.hasNext(SubtypeIterator.java:28)
            	at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:56)
            	at hudson.init.InitializerFinder.discoverTasks(InitializerFinder.java:33)
            	at hudson.PluginManager.dynamicLoad(PluginManager.java:479)
            	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1318)
            	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1121)
            	at …
            

            Seems to be a class loading order problem: this code loads Map.Entry for the first time (seems impossible!), which in turns triggers class loading, which the JVM rejects since we are already loading a class.

            jglick Jesse Glick added a comment - Interesting…after having run 1.606, created the empty logger, deleted it , I tried to install Support Core (2.21) in the same session and got linkage errors, for example java.lang.ClassCircularityError: java/util/Map$Entry at com.cloudbees.jenkins.support.impl.JenkinsLogs$CustomHandler.publish(JenkinsLogs.java:506) at java.util.logging.Logger.log(Logger.java:738) at java.util.logging.Logger.doLog(Logger.java:765) at java.util.logging.Logger.log(Logger.java:788) at org.eclipse.jetty.util.log.JavaUtilLog.debug(JavaUtilLog.java:108) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:439) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at hudson.util.MaskingClassLoader.loadClass(MaskingClassLoader.java:62) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1397) at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1074) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2693) at java.lang.Class.getDeclaredMethods(Class.java:1967) at org.jvnet.hudson.annotation_indexer.Index$2$1.fetch(Index.java:102) at org.jvnet.hudson.annotation_indexer.Index$2$1.hasNext(Index.java:72) at org.jvnet.hudson.annotation_indexer.SubtypeIterator.fetch(SubtypeIterator.java:18) at org.jvnet.hudson.annotation_indexer.SubtypeIterator.hasNext(SubtypeIterator.java:28) at hudson.init.TaskMethodFinder.discoverTasks(TaskMethodFinder.java:56) at hudson.init.InitializerFinder.discoverTasks(InitializerFinder.java:33) at hudson.PluginManager.dynamicLoad(PluginManager.java:479) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1318) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1121) at … Seems to be a class loading order problem: this code loads Map.Entry for the first time (seems impossible!), which in turns triggers class loading, which the JVM rejects since we are already loading a class.
            jglick Jesse Glick added a comment -

            JENKINS-23442 shows a similar problem from a different Handler.

            jglick Jesse Glick added a comment - JENKINS-23442 shows a similar problem from a different Handler .
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-23442 [ JENKINS-23442 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick added a comment - - edited

            Easiest way to reproduce is to run 1.606 on a fresh $JENKINS_HOME, create a log recorder with an empty logger, shut down, ln -s …/support-core-plugin/target/test-classes/the.hpl to $JENKINS_HOME/plugins/support-core.jpl, and restart.

            Have a fix for that mode. The empty logger spews over 3Mb of content, but it works. Unfortunately it does not work when creating the logger after installing the plugin.

            And making a functional test for this is not going to work, I think—there is too little control over class loading. Could work as an acceptance test perhaps.

            jglick Jesse Glick added a comment - - edited Easiest way to reproduce is to run 1.606 on a fresh $JENKINS_HOME , create a log recorder with an empty logger, shut down, ln -s …/support-core-plugin/target/test-classes/the.hpl to $JENKINS_HOME/plugins/support-core.jpl , and restart. Have a fix for that mode. The empty logger spews over 3Mb of content, but it works. Unfortunately it does not work when creating the logger after installing the plugin. And making a functional test for this is not going to work, I think—there is too little control over class loading. Could work as an acceptance test perhaps.
            jglick Jesse Glick added a comment -

            Reproduced in an acceptance test, but fixing is tedious. I do not know of a robust trick for preloading all classes that might be needed for a function, without actually running it.

            In some sense the fault is in Jetty: ClassLoader implementations must be extremely conservative about calling into foreign code, and in particular they should not make any (synchronous) logging calls.

            jglick Jesse Glick added a comment - Reproduced in an acceptance test, but fixing is tedious. I do not know of a robust trick for preloading all classes that might be needed for a function, without actually running it. In some sense the fault is in Jetty: ClassLoader implementations must be extremely conservative about calling into foreign code, and in particular they should not make any (synchronous) logging calls.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 31 (Web Link)" [ 12901 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "ATH PR 24 (Web Link)" [ 12902 ]

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/com/cloudbees/jenkins/support/impl/JenkinsLogs.java
            http://jenkins-ci.org/commit/support-core-plugin/43f0f84f2eee04623e464e9761c46915af6fd2bb
            Log:
            [FIXED JENKINS-27669] Force classes used by CustomHandler.publish to be loaded before we actually receive any log records.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/com/cloudbees/jenkins/support/impl/JenkinsLogs.java http://jenkins-ci.org/commit/support-core-plugin/43f0f84f2eee04623e464e9761c46915af6fd2bb Log: [FIXED JENKINS-27669] Force classes used by CustomHandler.publish to be loaded before we actually receive any log records.
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/com/cloudbees/jenkins/support/impl/JenkinsLogs.java
            http://jenkins-ci.org/commit/support-core-plugin/62ee0fdfa3325f121858644d6837312d2935cf57
            Log:
            Merge pull request #31 from jglick/ClassCircularityError-JENKINS-27669

            JENKINS-27669 Prevent ClassCircularityError

            Compare: https://github.com/jenkinsci/support-core-plugin/compare/db3e7f4e42ef...62ee0fdfa332

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/com/cloudbees/jenkins/support/impl/JenkinsLogs.java http://jenkins-ci.org/commit/support-core-plugin/62ee0fdfa3325f121858644d6837312d2935cf57 Log: Merge pull request #31 from jglick/ClassCircularityError- JENKINS-27669 JENKINS-27669 Prevent ClassCircularityError Compare: https://github.com/jenkinsci/support-core-plugin/compare/db3e7f4e42ef...62ee0fdfa332

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/test/java/plugins/SupportCorePluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/64c367031d50a70afcd9ea37c17a5de164f827c8
            Log:
            JENKINS-27669 Demonstrate ClassCircularityError before fix.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/plugins/SupportCorePluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/64c367031d50a70afcd9ea37c17a5de164f827c8 Log: JENKINS-27669 Demonstrate ClassCircularityError before fix.

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/test/java/plugins/SupportCorePluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/6f66161fbde34a33942b83ce642b5d7f747a4203
            Log:
            Merge pull request #24 from jglick/ClassCircularityError-JENKINS-27669

            JENKINS-27669 Demonstrate ClassCircularityError before fix

            Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/2920150fe326...6f66161fbde3

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/test/java/plugins/SupportCorePluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/6f66161fbde34a33942b83ce642b5d7f747a4203 Log: Merge pull request #24 from jglick/ClassCircularityError- JENKINS-27669 JENKINS-27669 Demonstrate ClassCircularityError before fix Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/2920150fe326...6f66161fbde3
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 161917 ] JNJira + In-Review [ 196897 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CE-526 (Web Link)" [ 19168 ]
            dnusbaum Devin Nusbaum made changes -
            Link This issue relates to JENKINS-55014 [ JENKINS-55014 ]
            aheritier Arnaud Héritier made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Link This issue relates to JENKINS-62472 [ JENKINS-62472 ]

            People

              jglick Jesse Glick
              ydubreuil Yoann Dubreuil
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: