• Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Trivial Trivial
    • _unsorted
    • None
    • jenkins 2.319.1 and 2.319.2
      test jenkins on centos7.9,
      prod jenkins on RHEL7.9
      tried with java 1.8.312 and 11.0.13.0.8

      After upgrading from jenkins v2.303.3 to the latest LTS version 2.319.2 jenkins service hangs during startup.

      after few minutes I receive an error

      [root@**** jenkins]$ systemctl start jenkins
      Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

      but it looks like jenkins instance itself starts up normally, I can login to ui, update plugins, etc...

      I have 2 instances of jenkins and have similar behavior on the both servers.

      On prod jenkins instance there is no errors in log, on test  I have one.

      All plugins are up to date.

      I had to restore backup and revert to 2.303.3 version.

      Logs I'll attach soon.

          [JENKINS-67632] Jenkins service hang during startup

          Andrii Malyi added a comment - - edited

          using startup script and  java 11

          [root@dev-jenk01 init.d]$ ./jenkins start
          Starting Jenkins WARNING: An illegal reflective access operation has occurred
          WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
          WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
          WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
          WARNING: All illegal access operations will be denied in a future release

           

          with java 8 there is  no error in log file

          Running from: /usr/lib/jenkins/jenkins.war
          2022-01-19 17:59:19.527+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated
          2022-01-19 17:59:19.566+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated
          2022-01-19 17:59:19.582+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @516ms to org.eclipse.jetty.util.log.JavaUtilLog
          2022-01-19 17:59:19.638+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
          2022-01-19 17:59:19.660+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
          2022-01-19 17:59:19.735+0000 [id=1] INFO winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]
          2022-01-19 17:59:19.766+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.
          13+8-LTS
          2022-01-19 17:59:20.114+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
          2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
          2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
          2022-01-19 17:59:20.159+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
          2022-01-19 17:59:20.727+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
          2022-01-19 17:59:20.932+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@7f811d00{Jenkins v2.319.2,/,file:///var/cache/jenkins/war/,AVAILABLE}

          {/var/cache/jenkins/war}

          2022-01-19 17:59:20.941+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@54107f42(root,h=[*********** infrastructure root ca],a=[],w=[]) for Server@6cea706c[provider=null
          ,keyStore=null,trustStore=null]
          2022-01-19 17:59:20.942+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@3bd7f8dc(jenkins,h=[inf-jenk01.*****, ********],a=[],w=[]) for Server@6cea706c[provider=null
          ,keyStore=null,trustStore=null]
          2022-01-19 17:59:21.227+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@71d15f18{SSL, (ssl, http/1.1)}

          {0.0.0.0:8443}

          2022-01-19 17:59:21.227+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @2161ms
          2022-01-19 17:59:21.227+0000 [id=25] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
          2022-01-19 17:59:21.448+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
          2022-01-19 17:59:21.875+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          2022-01-19 17:59:22.265+0000 [id=31] INFO o.j.h.p.backup.BackupPluginImpl#loadConfiguration: Loading configuration...
          2022-01-19 17:59:28.318+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          2022-01-19 17:59:28.357+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
          2022-01-19 17:59:29.547+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          2022-01-19 17:59:30.077+0000 [id=29] WARNING o.j.p.d.DockerBuilder$DescriptorImpl#<init>: Docker URL is not set, docker client won't be initialized
          2022-01-19 17:59:30.972+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
          2022-01-19 17:59:31.637+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
          2022-01-19 17:59:31.647+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
          2022-01-19 17:59:31.655+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
          2022-01-19 17:59:31.739+0000 [id=28] INFO o.s.s.l.DefaultSpringSecurityContextSource#<init>: URL 'ldaps://ldap.****/dc=****', root DN is 'dc=***,dc=***'
          2022-01-19 17:59:31.770+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
          2022-01-19 17:59:31.796+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 7 ms
          2022-01-19 17:59:31.930+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
          2022-01-19 17:59:31.954+0000 [id=22] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running

          Andrii Malyi added a comment - - edited using startup script and  java 11 [root@dev-jenk01 init.d] $ ./jenkins start Starting Jenkins WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 ( file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar ) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release   with java 8 there is  no error in log file Running from: /usr/lib/jenkins/jenkins.war 2022-01-19 17:59:19.527+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated 2022-01-19 17:59:19.566+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated 2022-01-19 17:59:19.582+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @516ms to org.eclipse.jetty.util.log.JavaUtilLog 2022-01-19 17:59:19.638+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file 2022-01-19 17:59:19.660+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2022-01-19 17:59:19.735+0000 [id=1] INFO winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$] 2022-01-19 17:59:19.766+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0. 13+8-LTS 2022-01-19 17:59:20.114+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0 2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults 2022-01-19 17:59:20.159+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms 2022-01-19 17:59:20.727+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME") 2022-01-19 17:59:20.932+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@7f811d00{Jenkins v2.319.2,/, file:///var/cache/jenkins/war/,AVAILABLE } {/var/cache/jenkins/war} 2022-01-19 17:59:20.941+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@54107f42(root,h= [*********** infrastructure root ca] ,a=[],w=[]) for Server@6cea706c[provider=null ,keyStore=null,trustStore=null] 2022-01-19 17:59:20.942+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@3bd7f8dc(jenkins,h= [inf-jenk01.*****, ********] ,a=[],w=[]) for Server@6cea706c[provider=null ,keyStore=null,trustStore=null] 2022-01-19 17:59:21.227+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@71d15f18{SSL, (ssl, http/1.1)} {0.0.0.0:8443} 2022-01-19 17:59:21.227+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @2161ms 2022-01-19 17:59:21.227+0000 [id=25] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled 2022-01-19 17:59:21.448+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 2022-01-19 17:59:21.875+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2022-01-19 17:59:22.265+0000 [id=31] INFO o.j.h.p.backup.BackupPluginImpl#loadConfiguration: Loading configuration... 2022-01-19 17:59:28.318+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 2022-01-19 17:59:28.357+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 2022-01-19 17:59:29.547+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 2022-01-19 17:59:30.077+0000 [id=29] WARNING o.j.p.d.DockerBuilder$DescriptorImpl#<init>: Docker URL is not set, docker client won't be initialized 2022-01-19 17:59:30.972+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded 2022-01-19 17:59:31.637+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted 2022-01-19 17:59:31.647+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 2022-01-19 17:59:31.655+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated 2022-01-19 17:59:31.739+0000 [id=28] INFO o.s.s.l.DefaultSpringSecurityContextSource#<init>: URL 'ldaps://ldap.**** /dc= **** ', root DN is 'dc= *** ,dc= ***' 2022-01-19 17:59:31.770+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata 2022-01-19 17:59:31.796+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 7 ms 2022-01-19 17:59:31.930+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization 2022-01-19 17:59:31.954+0000 [id=22] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running

          Basil Crow added a comment -

          Your Jenkins log contains numerous errors like this:

           

          2022-01-18 11:47:11.006+0000 [id=30]    WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues
          java.lang.RuntimeException: unsupported Java version: 11
                  at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1674)
                  at org.jruby.RubyInstanceConfig.<clinit>(RubyInstanceConfig.java:1387)
          Caused: java.lang.ExceptionInInitializerError
                  at org.jruby.embed.internal.AbstractLocalContextProvider.<init>(AbstractLocalContextProvider.java:42)
                  at org.jruby.embed.internal.SingleThreadLocalContextProvider.<init>(SingleThreadLocalContextProvider.java:43)
                  at org.jruby.embed.ScriptingContainer.getProviderInstance(ScriptingContainer.java:242)
                  at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:226)
                  at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:192)
                  at org.kohsuke.stapler.jelly.jruby.JRubyFacet.<init>(JRubyFacet.java:65)
                  at ruby.RubyRuntimePlugin.registerJRubyFacet(RubyRuntimePlugin.java:39)
                  at ruby.RubyRuntimePlugin.start(RubyRuntimePlugin.java:30)
                  at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:410)
                  at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:399)
          Caused: java.io.IOException: Failed to initialize
                  at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:402)
                  at hudson.PluginManager$2$1$1.run(PluginManager.java:536)
                  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
                  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
                  at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151)
                  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
                  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
                  at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
                  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
                  at java.base/java.lang.Thread.run(Thread.java:829)
          

          Please read this blog post.

          Basil Crow added a comment - Your Jenkins log contains numerous errors like this:   2022-01-18 11:47:11.006+0000 [id=30] WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues java.lang.RuntimeException: unsupported Java version: 11 at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1674) at org.jruby.RubyInstanceConfig.<clinit>(RubyInstanceConfig.java:1387) Caused: java.lang.ExceptionInInitializerError at org.jruby.embed.internal.AbstractLocalContextProvider.<init>(AbstractLocalContextProvider.java:42) at org.jruby.embed.internal.SingleThreadLocalContextProvider.<init>(SingleThreadLocalContextProvider.java:43) at org.jruby.embed.ScriptingContainer.getProviderInstance(ScriptingContainer.java:242) at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:226) at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:192) at org.kohsuke.stapler.jelly.jruby.JRubyFacet.<init>(JRubyFacet.java:65) at ruby.RubyRuntimePlugin.registerJRubyFacet(RubyRuntimePlugin.java:39) at ruby.RubyRuntimePlugin.start(RubyRuntimePlugin.java:30) at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:410) at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:399) Caused: java.io.IOException: Failed to initialize at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:402) at hudson.PluginManager$2$1$1.run(PluginManager.java:536) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Please read this blog post .

          Andrii Malyi added a comment -

          basil , Thanks for the information.

          For now, I've already ruled out plugins as possible causes of the reported behavior. I created a clone of virtual machine and in this "sandbox" I removed all jenkins plugins, I observe the same behavior. Now I'm trying to figure out what's wrong with clean jenkins w\o plugins.

          BTW, as I mentioned above, I tried out 2 configuration - with Java 1.8 and Java 11. There are no errors on java 1.8, and there are some errors on java 11 (ruby plugin ,illegal reflective access*, etc) but it looks like it doesn’t matter because behavior is the same for any version of Java machine.

          Andrii Malyi added a comment - basil , Thanks for the information. For now, I've already ruled out plugins as possible causes of the reported behavior. I created a clone of virtual machine and in this "sandbox" I removed all jenkins plugins, I observe the same behavior. Now I'm trying to figure out what's wrong with clean jenkins w\o plugins. BTW, as I mentioned above, I tried out 2 configuration - with Java 1.8 and Java 11. There are no errors on java 1.8, and there are some errors on java 11 (ruby plugin ,illegal reflective access*, etc) but it looks like it doesn’t matter because behavior is the same for any version of Java machine.

          Basil Crow added a comment -

          If you're getting a hang and nothing is printed to the logs, try running jstack against the Java process and provide the thread dump here. Ideally, run it several times over the course of a few minutes to get a sample, and see if there is a thread that is stuck in the same stack trace throughout all samples.

          Basil Crow added a comment - If you're getting a hang and nothing is printed to the logs, try running jstack against the Java process and provide the thread dump here. Ideally, run it several times over the course of a few minutes to get a sample, and see if there is a thread that is stuck in the same stack trace throughout all samples.

          Andrii Malyi added a comment -

          Sorry guys.
          The resolution wasn't so deep. The problem was with updating the startup script.
          It is still not clear why the installation of the daemonize package and jenkins was successful, but the startup script was not changed properly.
          We found these differences in a detailed comparison with a newly installed instance.
          Now all works fine. Thanks.

          Andrii Malyi added a comment - Sorry guys. The resolution wasn't so deep. The problem was with updating the startup script. It is still not clear why the installation of the daemonize package and jenkins was successful, but the startup script was not changed properly. We found these differences in a detailed comparison with a newly installed instance. Now all works fine. Thanks.

          Dana Shankar added a comment -

          Jenkins service hangs after upgrading to 2.319.3 on Linux servers.

          Dana Shankar added a comment - Jenkins service hangs after upgrading to 2.319.3 on Linux servers.

          Mark Waite added a comment -

          dshanks your comment

          Jenkins service hangs after upgrading to 2.319.3 on Linux servers.

          lacks sufficient context about your environment for anyone to help you.

          The comment from andrews indicates that the problem originally described was resolved by local changes they made in their installation. You'll need to describe your situation in sufficient detail so that others can duplicate the problem or accept that the issue that you've reopened will be closed because andrews noted that the issue as reported has been resolved.

          Refer to https://www.jenkins.io/participate/report-issue/ for guidelines on the information that you need to provide in order for others to be able to help you.

          Mark Waite added a comment - dshanks your comment Jenkins service hangs after upgrading to 2.319.3 on Linux servers. lacks sufficient context about your environment for anyone to help you. The comment from andrews indicates that the problem originally described was resolved by local changes they made in their installation. You'll need to describe your situation in sufficient detail so that others can duplicate the problem or accept that the issue that you've reopened will be closed because andrews noted that the issue as reported has been resolved. Refer to https://www.jenkins.io/participate/report-issue/ for guidelines on the information that you need to provide in order for others to be able to help you.

          Mark Waite added a comment -

          Closing after one week with no reply from dshanks

          Mark Waite added a comment - Closing after one week with no reply from dshanks

            andrews Andrii Malyi
            andrews Andrii Malyi
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: