-
Bug
-
Resolution: Not A Defect
-
Trivial
-
None
-
jenkins 2.319.1 and 2.319.2
test jenkins on centos7.9,
prod jenkins on RHEL7.9
tried with java 1.8.312 and 11.0.13.0.8
After upgrading from jenkins v2.303.3 to the latest LTS version 2.319.2 jenkins service hangs during startup.
after few minutes I receive an error
[root@**** jenkins]$ systemctl start jenkins
Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.
but it looks like jenkins instance itself starts up normally, I can login to ui, update plugins, etc...
I have 2 instances of jenkins and have similar behavior on the both servers.
On prod jenkins instance there is no errors in log, on test I have one.
All plugins are up to date.
I had to restore backup and revert to 2.303.3 version.
Logs I'll attach soon.
[JENKINS-67632] Jenkins service hang during startup
Andrii Malyi
created issue -
Andrii Malyi
made changes -
| Attachment | New: jenkins.log-20220119.gz [ 57115 ] |
Andrii Malyi
made changes -
| Component/s | New: _unsorted [ 19622 ] | |
| Component/s | Original: ruby-runtime [ 15982 ] |
Andrii Malyi
made changes -
| Description |
Original:
After upgrading from jenkins v2.303.3 to the latest LTS version 2.319.2 jenkins service hangs during startup.
after few minutes I receive an error [root@**** jenkins]$ systemctl start jenkins Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details. but it looks like jenkins instance itself starts up normally, I can login to ui, update plugins, etc... I have 2 instances of jenkins and have similar behavior on the both servers. On prod jenkins instance there is no errors in log, on test I have one. I had to restore backup and revert to 2.303.3 version. Logs I'll attach soon. |
New:
After upgrading from jenkins v2.303.3 to the latest LTS version 2.319.2 jenkins service hangs during startup.
after few minutes I receive an error [root@**** jenkins]$ systemctl start jenkins Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details. but it looks like jenkins instance itself starts up normally, I can login to ui, update plugins, etc... I have 2 instances of jenkins and have similar behavior on the both servers. On prod jenkins instance there is no errors in log, on test I have one. All plugins are up to date. I had to restore backup and revert to 2.303.3 version. Logs I'll attach soon. |
Andrii Malyi
made changes -
| Priority | Original: Minor [ 4 ] | New: Trivial [ 5 ] |
Andrii Malyi
made changes -
| Priority | Original: Trivial [ 5 ] | New: Major [ 3 ] |
Basil Crow
added a comment - Your Jenkins log contains numerous errors like this:
2022-01-18 11:47:11.006+0000 [id=30] WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues
java.lang.RuntimeException: unsupported Java version: 11
at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1674)
at org.jruby.RubyInstanceConfig.<clinit>(RubyInstanceConfig.java:1387)
Caused: java.lang.ExceptionInInitializerError
at org.jruby.embed.internal.AbstractLocalContextProvider.<init>(AbstractLocalContextProvider.java:42)
at org.jruby.embed.internal.SingleThreadLocalContextProvider.<init>(SingleThreadLocalContextProvider.java:43)
at org.jruby.embed.ScriptingContainer.getProviderInstance(ScriptingContainer.java:242)
at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:226)
at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:192)
at org.kohsuke.stapler.jelly.jruby.JRubyFacet.<init>(JRubyFacet.java:65)
at ruby.RubyRuntimePlugin.registerJRubyFacet(RubyRuntimePlugin.java:39)
at ruby.RubyRuntimePlugin.start(RubyRuntimePlugin.java:30)
at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:410)
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:399)
Caused: java.io.IOException: Failed to initialize
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:402)
at hudson.PluginManager$2$1$1.run(PluginManager.java:536)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Please read this blog post.
Basil Crow
added a comment - Your Jenkins log contains numerous errors like this:
2022-01-18 11:47:11.006+0000 [id=30] WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues
java.lang.RuntimeException: unsupported Java version: 11
at org.jruby.RubyInstanceConfig.initGlobalJavaVersion(RubyInstanceConfig.java:1674)
at org.jruby.RubyInstanceConfig.<clinit>(RubyInstanceConfig.java:1387)
Caused: java.lang.ExceptionInInitializerError
at org.jruby.embed.internal.AbstractLocalContextProvider.<init>(AbstractLocalContextProvider.java:42)
at org.jruby.embed.internal.SingleThreadLocalContextProvider.<init>(SingleThreadLocalContextProvider.java:43)
at org.jruby.embed.ScriptingContainer.getProviderInstance(ScriptingContainer.java:242)
at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:226)
at org.jruby.embed.ScriptingContainer.<init>(ScriptingContainer.java:192)
at org.kohsuke.stapler.jelly.jruby.JRubyFacet.<init>(JRubyFacet.java:65)
at ruby.RubyRuntimePlugin.registerJRubyFacet(RubyRuntimePlugin.java:39)
at ruby.RubyRuntimePlugin.start(RubyRuntimePlugin.java:30)
at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:410)
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:399)
Caused: java.io.IOException: Failed to initialize
at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:402)
at hudson.PluginManager$2$1$1.run(PluginManager.java:536)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1151)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Please read this blog post . Andrii Malyi
added a comment - basil , Thanks for the information.
For now, I've already ruled out plugins as possible causes of the reported behavior. I created a clone of virtual machine and in this "sandbox" I removed all jenkins plugins, I observe the same behavior. Now I'm trying to figure out what's wrong with clean jenkins w\o plugins.
BTW, as I mentioned above, I tried out 2 configuration - with Java 1.8 and Java 11. There are no errors on java 1.8, and there are some errors on java 11 (ruby plugin ,illegal reflective access*, etc) but it looks like it doesn’t matter because behavior is the same for any version of Java machine.
Andrii Malyi
added a comment - basil , Thanks for the information.
For now, I've already ruled out plugins as possible causes of the reported behavior. I created a clone of virtual machine and in this "sandbox" I removed all jenkins plugins, I observe the same behavior. Now I'm trying to figure out what's wrong with clean jenkins w\o plugins.
BTW, as I mentioned above, I tried out 2 configuration - with Java 1.8 and Java 11. There are no errors on java 1.8, and there are some errors on java 11 (ruby plugin ,illegal reflective access*, etc) but it looks like it doesn’t matter because behavior is the same for any version of Java machine. Basil Crow
added a comment - If you're getting a hang and nothing is printed to the logs, try running jstack against the Java process and provide the thread dump here. Ideally, run it several times over the course of a few minutes to get a sample, and see if there is a thread that is stuck in the same stack trace throughout all samples.
Basil Crow
added a comment - If you're getting a hang and nothing is printed to the logs, try running jstack against the Java process and provide the thread dump here. Ideally, run it several times over the course of a few minutes to get a sample, and see if there is a thread that is stuck in the same stack trace throughout all samples.
using startup script and java 11
[root@dev-jenk01 init.d]$ ./jenkins start
Starting Jenkins WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
with java 8 there is no error in log file
Running from: /usr/lib/jenkins/jenkins.war
{/var/cache/jenkins/war}2022-01-19 17:59:19.527+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMax is now deprecated
2022-01-19 17:59:19.566+0000 [id=1] WARNING winstone.Logger#logInternal: Parameter handlerCountMaxIdle is now deprecated
2022-01-19 17:59:19.582+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @516ms to org.eclipse.jetty.util.log.JavaUtilLog
2022-01-19 17:59:19.638+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2022-01-19 17:59:19.660+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2022-01-19 17:59:19.735+0000 [id=1] INFO winstone.Logger#logInternal: Exclude Ciphers [^.*_(MD5|SHA|SHA1)$, ^TLS_RSA_.*$, ^SSL_.*$, ^.*_NULL_.*$, ^.*_anon_.*$]
2022-01-19 17:59:19.766+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.
13+8-LTS
2022-01-19 17:59:20.114+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2022-01-19 17:59:20.158+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2022-01-19 17:59:20.159+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2022-01-19 17:59:20.727+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
2022-01-19 17:59:20.932+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@7f811d00{Jenkins v2.319.2,/,file:///var/cache/jenkins/war/,AVAILABLE}
2022-01-19 17:59:20.941+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@54107f42(root,h=[*********** infrastructure root ca],a=[],w=[]) for Server@6cea706c[provider=null
{0.0.0.0:8443},keyStore=null,trustStore=null]
2022-01-19 17:59:20.942+0000 [id=1] INFO o.e.j.util.ssl.SslContextFactory#load: x509=X509@3bd7f8dc(jenkins,h=[inf-jenk01.*****, ********],a=[],w=[]) for Server@6cea706c[provider=null
,keyStore=null,trustStore=null]
2022-01-19 17:59:21.227+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@71d15f18{SSL, (ssl, http/1.1)}
2022-01-19 17:59:21.227+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @2161ms
2022-01-19 17:59:21.227+0000 [id=25] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2022-01-19 17:59:21.448+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2022-01-19 17:59:21.875+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2022-01-19 17:59:22.265+0000 [id=31] INFO o.j.h.p.backup.BackupPluginImpl#loadConfiguration: Loading configuration...
2022-01-19 17:59:28.318+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2022-01-19 17:59:28.357+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2022-01-19 17:59:29.547+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2022-01-19 17:59:30.077+0000 [id=29] WARNING o.j.p.d.DockerBuilder$DescriptorImpl#<init>: Docker URL is not set, docker client won't be initialized
2022-01-19 17:59:30.972+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2022-01-19 17:59:31.637+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2022-01-19 17:59:31.647+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2022-01-19 17:59:31.655+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2022-01-19 17:59:31.739+0000 [id=28] INFO o.s.s.l.DefaultSpringSecurityContextSource#<init>: URL 'ldaps://ldap.****/dc=****', root DN is 'dc=***,dc=***'
2022-01-19 17:59:31.770+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2022-01-19 17:59:31.796+0000 [id=55] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Finished Download metadata. 7 ms
2022-01-19 17:59:31.930+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization
2022-01-19 17:59:31.954+0000 [id=22] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running