-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
Hi,
We recently starting using Audit Trail plugin and enabled Elastic Logger after which our Jenkins is not able to restart. I tried reading logs and it shows nothing just jenkins timing out while starting and even after setting the timeout value to 15 minutes , it is still not able to restart.
I changed the Logger to File Logger and now it is working fine.
Can you please help in identifying the issue with the elastic logger ?
Jenkins Version : - 2.414.1
Audit Trail Plugin :- 361.v82cde86c784e
[JENKINS-72939] Audit Trail : Jenkins not able to restart after enabling ES logger
Aakash
added a comment - From timing out , I meant like below where it is failing without any errors and just timing out. I will get you the thread dumps in some time.
Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][jenkins.InitReactorRunner$1 onAttained] Started all plugins Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][jenkins.InitReactorRunner$1 onAttained] Augmented all extensions Mar 28 11:26:32 jenkins[17754]: WARNING: An illegal reflective access operation has occurred Mar 28 11:26:32 jenkins[17754]: WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/lib/jenkins/%25C/jenkins/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) Mar 28 11:26:32 jenkins[17754]: WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1 Mar 28 11:26:32 jenkins[17754]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Mar 28 11:26:32 jenkins[17754]: WARNING: All illegal access operations will be denied in a future release Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][org.marvelution.jji.tunnel.TunnelManager loadNgrokNativeLibrary] Created classloader to support tunneling using file:/var/lib/jenkins/plugins/jira-integration/WEB-INF/ngrok/ngrok-java-native-linux-x86_64.jar Mar 28 11:26:33 jenkins[17754]: [2024-03-28 11:26:33][WARNING][org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval load] There are 110 deprecated approved script hashes and 0 deprecated approved classpath hashes. They will be rehashed upon next use and that may cause performance issues until all of them are converted or removed. Mar 28 11:26:33 jenkins[17754]: [2024-03-28 11:26:33][INFO ][hudson.plugins.build_timeout.global.GlobalTimeOutConfiguration load] global timeout not set Mar 28 11:27:53 systemd[1]: jenkins.service start operation timed out. Terminating. Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][winstone.Logger logInternal] JVM is terminating. Shutting down Jetty Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][org.eclipse.jetty.server.Server doStop] Stopped Server@7133da86{STOPPING}[10.0.15,sto=0] Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][org.eclipse.jetty.server.AbstractConnector doStop] Stopped ServerConnector@15bbf42f{SSL, (ssl, http/1.1)}{****:8444} Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][hudson.lifecycle.Lifecycle onStatusUpdate] Stopping Jenkins Mar 28 11:29:23 systemd[1]: jenkins.service stop-sigterm timed out. Killing. Mar 28 11:29:23 systemd[1]: jenkins.service: main process exited, code=killed, status=9/KILL Mar 28 11:29:23 systemd[1]: Failed to start Jenkins Continuous Integration Server.
Aakash
added a comment - From timing out , I meant like below where it is failing without any errors and just timing out. I will get you the thread dumps in some time.
Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][jenkins.InitReactorRunner$1 onAttained] Started all plugins
Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][jenkins.InitReactorRunner$1 onAttained] Augmented all extensions
Mar 28 11:26:32 jenkins[17754]: WARNING: An illegal reflective access operation has occurred
Mar 28 11:26:32 jenkins[17754]: WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/ var /lib/jenkins/%25C/jenkins/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang. Class , int )
Mar 28 11:26:32 jenkins[17754]: WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
Mar 28 11:26:32 jenkins[17754]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 28 11:26:32 jenkins[17754]: WARNING: All illegal access operations will be denied in a future release
Mar 28 11:26:32 jenkins[17754]: [2024-03-28 11:26:32][INFO ][org.marvelution.jji.tunnel.TunnelManager loadNgrokNativeLibrary] Created classloader to support tunneling using file:/ var /lib/jenkins/plugins/jira-integration/WEB-INF/ngrok/ngrok-java- native -linux-x86_64.jar
Mar 28 11:26:33 jenkins[17754]: [2024-03-28 11:26:33][WARNING][org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval load] There are 110 deprecated approved script hashes and 0 deprecated approved classpath hashes. They will be rehashed upon next use and that may cause performance issues until all of them are converted or removed.
Mar 28 11:26:33 jenkins[17754]: [2024-03-28 11:26:33][INFO ][hudson.plugins.build_timeout.global.GlobalTimeOutConfiguration load] global timeout not set
Mar 28 11:27:53 systemd[1]: jenkins.service start operation timed out. Terminating.
Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][winstone.Logger logInternal] JVM is terminating. Shutting down Jetty
Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][org.eclipse.jetty.server.Server doStop] Stopped Server@7133da86{STOPPING}[10.0.15,sto=0]
Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][org.eclipse.jetty.server.AbstractConnector doStop] Stopped ServerConnector@15bbf42f{SSL, (ssl, http/1.1)}{****:8444}
Mar 28 11:27:53 jenkins[17754]: [2024-03-28 11:27:53][INFO ][hudson.lifecycle.Lifecycle onStatusUpdate] Stopping Jenkins
Mar 28 11:29:23 systemd[1]: jenkins.service stop-sigterm timed out. Killing.
Mar 28 11:29:23 systemd[1]: jenkins.service: main process exited, code=killed, status=9/KILL
Mar 28 11:29:23 systemd[1]: Failed to start Jenkins Continuous Integration Server.
Closing for inactivity, please reopen in case you have thread dumps to share.
abalyan2395 I'm unsure I understand what " it shows nothing just jenkins timing out " means. Assuming it hangs, then you want to capture a couple of thread dumps at startup to understand exactly what the instance is doing.