Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72968

Jenkins initialization thread/52 died

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core
    • Ubuntu 24.04
      openjdk 21.0.2
      jenkins 2.440.2

      Hello,

      I've got a freshly installed Ubuntu 24.04 with 21.0.2 openjdk.
      Sys has 32gb ram, 16 core, 500 gb storage.
      jenkins user already exist.
      Tried to install Jenkins via 'apt install jenkins' but its failed at ~60%. 

      rerun says:

      root@jenkins:/home/jenkins# apt install jenkins
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
jenkins is already the newest version (2.440.2).

      listening on port, but hanging..

      x.x.x.x:8080/login?from=%2F

'Please wait while Jenkins is getting ready to work ...'

      systemctl status:

       root@jenkins:/home/jenkins# systemctl status jenkins
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)
     Active: activating (start) since Thu 2024-04-04 15:13:20 UTC; 4min 7s ago
   Main PID: 28509 (java)
      Tasks: 102 (limit: 38470)
     Memory: 267.1M (peak: 268.9M)
        CPU: 11.536s
     CGroup: /system.slice/jenkins.service
             └─28509 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************
Apr 04 15:13:23 jenkins jenkins[28509]: Jenkins initial setup is required. An admin user has been created and a password generated.
Apr 04 15:13:23 jenkins jenkins[28509]: Please use the following password to proceed to installation:
Apr 04 15:13:23 jenkins jenkins[28509]: e4512fd73337478fa73f266adc7c7846
Apr 04 15:13:23 jenkins jenkins[28509]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************
Apr 04 15:13:23 jenkins jenkins[28509]: *************************************************************

      journalctl:

      Apr 04 15:08:20 jenkins systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 28.
Apr 04 15:08:20 jenkins systemd[1]: Starting jenkins.service - Jenkins Continuous Integration Server...
Apr 04 15:08:20 jenkins jenkins[28251]: Running from: /usr/share/java/jenkins.war
Apr 04 15:08:20 jenkins jenkins[28251]: 2024-04-04 15:08:20.536+0000 [id=1]        INFO        winstone.Logger#logInternal: Beginning extraction from war file
Apr 04 15:08:20 jenkins jenkins[28251]: 2024-04-04 15:08:20.622+0000 [id=1]        WARNING        o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
Apr 04 15:08:20 jenkins jenkins[28251]: 2024-04-04 15:08:20.689+0000 [id=1]        INFO        org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 2>
Apr 04 15:08:20 jenkins jenkins[28251]: 2024-04-04 15:08:20.934+0000 [id=1]        INFO        o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Apr 04 15:08:20 jenkins jenkins[28251]: 2024-04-04 15:08:20.982+0000 [id=1]        INFO        o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.364+0000 [id=1]        INFO        hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.456+0000 [id=1]        INFO        o.e.j.s.handler.ContextHandler#doStart: Started w.@7d373bcf{Jenkins v2.440.2,/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/w>
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.466+0000 [id=1]        INFO        o.e.j.server.AbstractConnector#doStart: Started ServerConnector@45afc369{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.475+0000 [id=1]        INFO        org.eclipse.jetty.server.Server#doStart: Started Server@f0f2775{STARTING}[10.0.20,sto=0] @1363ms
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.476+0000 [id=55]        INFO        winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.717+0000 [id=64]        INFO        jenkins.InitReactorRunner$1#onAttained: Started initialization
Apr 04 15:08:21 jenkins jenkins[28251]: 2024-04-04 15:08:21.720+0000 [id=85]        INFO        jenkins.InitReactorRunner$1#onAttained: Listed all plugins
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.493+0000 [id=95]        INFO        jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.498+0000 [id=65]        INFO        jenkins.InitReactorRunner$1#onAttained: Started all plugins
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.503+0000 [id=80]        INFO        jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.708+0000 [id=92]        INFO        jenkins.InitReactorRunner$1#onAttained: System config loaded
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.709+0000 [id=88]        INFO        jenkins.InitReactorRunner$1#onAttained: System config adapted
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.709+0000 [id=88]        INFO        jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.711+0000 [id=70]        INFO        jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.732+0000 [id=109]        INFO        hudson.util.Retrier#start: Attempt #1 to do the action check updates server
Apr 04 15:08:22 jenkins jenkins[28251]: 2024-04-04 15:08:22.765+0000 [id=90]        INFO        jenkins.install.SetupWizard#init:
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: Jenkins initial setup is required. An admin user has been created and a password generated.
Apr 04 15:08:22 jenkins jenkins[28251]: Please use the following password to proceed to installation:
Apr 04 15:08:22 jenkins jenkins[28251]: e4512fd73337478fa73f266adc7c7846
Apr 04 15:08:22 jenkins jenkins[28251]: This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: start operation timed out. Terminating.
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.320+0000 [id=56]        INFO        winstone.Logger#logInternal: JVM is terminating. Shutting down Jetty
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.322+0000 [id=56]        INFO        org.eclipse.jetty.server.Server#doStop: Stopped Server@f0f2775{STOPPING}[10.0.20,sto=0]
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.326+0000 [id=56]        INFO        o.e.j.server.AbstractConnector#doStop: Stopped ServerConnector@45afc369{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.329+0000 [id=56]        INFO        hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.337+0000 [id=56]        INFO        jenkins.model.Jenkins$16#onAttained: Started termination
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.345+0000 [id=56]        INFO        jenkins.model.Jenkins$16#onAttained: Completed termination
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.346+0000 [id=56]        INFO        jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.350+0000 [id=56]        INFO        jenkins.model.Jenkins#_cleanUpShutdownPluginManager: Stopping plugin manager
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.350+0000 [id=56]        INFO        jenkins.model.Jenkins#_cleanUpPersistQueue: Persisting build queue
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.362+0000 [id=56]        INFO        jenkins.model.Jenkins#_cleanUpAwaitDisconnects: Waiting for node disconnection completion
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.362+0000 [id=56]        INFO        hudson.lifecycle.Lifecycle#onStatusUpdate: Jenkins stopped
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.363+0000 [id=56]        INFO        hudson.WebAppMain#contextDestroyed: Shutting down a Jenkins instance that was still starting up
Apr 04 15:13:20 jenkins jenkins[28251]: java.lang.Throwable: reason
Apr 04 15:13:20 jenkins jenkins[28251]:         at hudson.WebAppMain.contextDestroyed(WebAppMain.java:407)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:1059)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:636)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.ContextHandler.contextDestroyed(ContextHandler.java:1016)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.servlet.ServletHandler.doStop(ServletHandler.java:306)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.security.SecurityHandler.doStop(SecurityHandler.java:412)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.security.ConstraintSecurityHandler.doStop(ConstraintSecurityHandler.java:413)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.session.SessionHandler.doStop(SessionHandler.java:497)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:1039)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:399)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1311)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1087)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:312)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:97)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.server.Server.doStop(Server.java:517)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Apr 04 15:13:20 jenkins jenkins[28251]:         at winstone.Launcher.shutdown(Launcher.java:442)
Apr 04 15:13:20 jenkins jenkins[28251]:         at winstone.ShutdownHook.run(ShutdownHook.java:28)
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.365+0000 [id=56]        INFO        o.e.j.s.handler.ContextHandler#doStop: Stopped w.@7d373bcf{Jenkins v2.440.2,/,null,STOPPED}{/var/cache/jenkins/war}
Apr 04 15:13:20 jenkins jenkins[28251]: 2024-04-04 15:13:20.365+0000 [id=52]        SEVERE        h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (Jenkins initialization thread/52) died >
Apr 04 15:13:20 jenkins jenkins[28251]: java.lang.ClassNotFoundException: hudson.util.HudsonFailedToLoad
Apr 04 15:13:20 jenkins jenkins[28251]:         at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.webapp.WebAppClassLoader.foundClass(WebAppClassLoader.java:594)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.webapp.WebAppClassLoader.loadAsResource(WebAppClassLoader.java:567)
Apr 04 15:13:20 jenkins jenkins[28251]:         at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:502)
Apr 04 15:13:20 jenkins jenkins[28251]:         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
Apr 04 15:13:20 jenkins jenkins[28251]: Caused: java.lang.NoClassDefFoundError: hudson/util/HudsonFailedToLoad
Apr 04 15:13:20 jenkins jenkins[28251]:         at hudson.WebAppMain$3.run(WebAppMain.java:276)
Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: Failed with result 'timeout'.
Apr 04 15:13:20 jenkins systemd[1]: Failed to start jenkins.service - Jenkins Continuous Integration Server.
Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: Consumed 10.006s CPU time.
Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 29.

      tried to collect jstack, attached to the ticket

      root@jenkins:/home/jenkins# sudo dpkg --configure -a
Setting up jenkins (2.440.2) ...
Job for jenkins.service failed because a timeout was exceeded.
See "systemctl status jenkins.service" and "journalctl -xeu jenkins.service" for details.
invoke-rc.d: initscript jenkins, action "start" failed.
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: timeout) since Thu 2024-04-04 16:23:44 UTC; 9ms ago
    Process: 36357 ExecStart=/usr/bin/jenkins (code=exited, status=143)
   Main PID: 36357 (code=exited, status=143)
     Status: "Jenkins stopped"
        CPU: 11.487s
dpkg: error processing package jenkins (--configure):
 installed jenkins package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 jenkins

       

      Could you please  check whats wrong with it?

      all ideas are appreciated.

       

      Br,

      Laszlo

        1. jenkins_v2.log
          30 kB
        2. jstack_11min.txt
          92 kB
        3. jstack_10min.txt
          92 kB
        4. jenkins.log
          12 kB
        5. jenkins.service
          5 kB
        6. jstack.txt
          106 kB

          [JENKINS-72968] Jenkins initialization thread/52 died

          L added a comment -

          just downgraded java to verify, it fails with 17 too

          root@jenkins:/home/jenkins# java --version
openjdk 17.0.10 2024-01-16

          L added a comment - just downgraded java to verify, it fails with 17 too root@jenkins:/home/jenkins# java --version openjdk 17.0.10 2024-01-16

          Basil Crow added a comment -

          The above shows that you haven't completed the setup wizard within the service timeout window of 90 seconds. You should be able to complete the setup wizard and then the service will finally finish starting. If you can't complete the wizard within 90 seconds, you can set TimeoutStartSec=infinity in the systemd configuration (sudo systemctl edit jenkins) to give yourself more time.

          Thinking about this some more, maybe we need to report successful service startup to systemd at the beginning of the setup wizard rather than the end to avoid scenarios like this. This shouldn't be a difficult change to make. However, I'm a little bit surprised, because this isn't a new behavior: systemd support has behaved this way since it was introduced several years ago. If this is really as annoying as this bug report would suggest, I wonder why more people haven't complained about it. Or maybe they have, and I just haven't seen the Jira ticket? I'd welcome opinions from other people about this. If it really is as annoying as this bug report would suggest, then it would be a simple change to make Jenkins report successful service startup at the beginning of the setup wizard to satisfy systemd.

          Basil Crow added a comment - The above shows that you haven't completed the setup wizard within the service timeout window of 90 seconds. You should be able to complete the setup wizard and then the service will finally finish starting. If you can't complete the wizard within 90 seconds, you can set TimeoutStartSec=infinity in the systemd configuration ( sudo systemctl edit jenkins ) to give yourself more time. Thinking about this some more, maybe we need to report successful service startup to systemd at the beginning of the setup wizard rather than the end to avoid scenarios like this. This shouldn't be a difficult change to make. However, I'm a little bit surprised, because this isn't a new behavior: systemd support has behaved this way since it was introduced several years ago. If this is really as annoying as this bug report would suggest, I wonder why more people haven't complained about it. Or maybe they have, and I just haven't seen the Jira ticket? I'd welcome opinions from other people about this. If it really is as annoying as this bug report would suggest, then it would be a simple change to make Jenkins report successful service startup at the beginning of the setup wizard to satisfy systemd.

          Mark Waite added a comment -

          I didn't realize that was the problem. There have been a number of reports of timeouts reported that might be explained by failure to complete the setup wizard promptly. I would very much like to have Jenkins report successful startup at the beginning of the setup wizard.

          Mark Waite added a comment - I didn't realize that was the problem. There have been a number of reports of timeouts reported that might be explained by failure to complete the setup wizard promptly. I would very much like to have Jenkins report successful startup at the beginning of the setup wizard.

          Basil Crow added a comment - - edited

          I think my analysis in the previous comment was incorrect, as I can't reproduce the scenario locally. When I clear my Jenkins home directory and restart Jenkins with systemctl restart jenkins, I get:

          Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:29 basil-desktop jenkins[2357448]: Jenkins initial setup is required. An admin user has been created and a password generated.
Apr 04 19:30:29 basil-desktop jenkins[2357448]: Please use the following password to proceed to installation:
Apr 04 19:30:29 basil-desktop jenkins[2357448]: 7bff17f56f6148459beaf3342f1db3cc
Apr 04 19:30:29 basil-desktop jenkins[2357448]: This may also be found at: /home/basil/jenkins_homes/test/secrets/initialAdminPassword
Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:29 basil-desktop jenkins[2357448]: *************************************************************
Apr 04 19:30:45 basil-desktop jenkins[2357448]: 2024-04-05 02:30:45.023+0000 [id=59]        INFO        jenkins.InitReactorRunner$1#onAttained: Completed initialization
Apr 04 19:30:45 basil-desktop jenkins[2357448]: 2024-04-05 02:30:45.033+0000 [id=39]        INFO        hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
Apr 04 19:30:45 basil-desktop systemd[1]: Started Jenkins Continuous Integration Server.

          At which point the systemctl restart jenkins command has returned and the service is online. And the "Jenkins is fully up and running" log statement happens about 15 seconds after I initiated the process, which is about how long Jenkins takes to come up on a clean home directory on my system, which is also when systemd is notified of successful service startup. So on my system at least, everything is working as intended.

          In the above logs, I see:

          Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:08:22 jenkins jenkins[28251]: *************************************************************
Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: start operation timed out. Terminating.

          So we never see the "Jenkins is fully up and running" and systemd never gets notified, and times out about 4 minutes later. That's an odd amount of time, since I thought the default timeout was 90 seconds. But anyway, the question is "what was Jenkins doing after it printed the last " *************************************************************" and the time that systemd gave up on it, during which it apparently didn't log anything.

          milkuci Are you sure you don't have JENKINS_LOG set to log somewhere other than systemd-journald? If so, you need to look there for logs instead of journalctl. And if you don't, then you'll need to run jstack during the hang period to see where Jenkins got stuck during the initialization process.

          Basil Crow added a comment - - edited I think my analysis in the previous comment was incorrect, as I can't reproduce the scenario locally. When I clear my Jenkins home directory and restart Jenkins with systemctl restart jenkins , I get: Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:29 basil-desktop jenkins[2357448]: Jenkins initial setup is required. An admin user has been created and a password generated. Apr 04 19:30:29 basil-desktop jenkins[2357448]: Please use the following password to proceed to installation: Apr 04 19:30:29 basil-desktop jenkins[2357448]: 7bff17f56f6148459beaf3342f1db3cc Apr 04 19:30:29 basil-desktop jenkins[2357448]: This may also be found at: /home/basil/jenkins_homes/test/secrets/initialAdminPassword Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:29 basil-desktop jenkins[2357448]: ************************************************************* Apr 04 19:30:45 basil-desktop jenkins[2357448]: 2024-04-05 02:30:45.023+0000 [id=59] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization Apr 04 19:30:45 basil-desktop jenkins[2357448]: 2024-04-05 02:30:45.033+0000 [id=39] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running Apr 04 19:30:45 basil-desktop systemd[1]: Started Jenkins Continuous Integration Server. At which point the systemctl restart jenkins command has returned and the service is online. And the "Jenkins is fully up and running" log statement happens about 15 seconds after I initiated the process, which is about how long Jenkins takes to come up on a clean home directory on my system, which is also when systemd is notified of successful service startup. So on my system at least, everything is working as intended. In the above logs, I see: Apr 04 15:08:22 jenkins jenkins[28251]: ************************************************************* Apr 04 15:08:22 jenkins jenkins[28251]: ************************************************************* Apr 04 15:08:22 jenkins jenkins[28251]: ************************************************************* Apr 04 15:13:20 jenkins systemd[1]: jenkins.service: start operation timed out. Terminating. So we never see the "Jenkins is fully up and running" and systemd never gets notified, and times out about 4 minutes later. That's an odd amount of time, since I thought the default timeout was 90 seconds. But anyway, the question is "what was Jenkins doing after it printed the last " *************************************************************" and the time that systemd gave up on it, during which it apparently didn't log anything. milkuci Are you sure you don't have JENKINS_LOG set to log somewhere other than systemd-journald ? If so, you need to look there for logs instead of journalctl . And if you don't, then you'll need to run jstack during the hang period to see where Jenkins got stuck during the initialization process.

          L added a comment -

          Basil thanks for the quick reply!
          Just increased to TimeoutStartSec=900.
          also just modified to logging into var/log

          Let me share the results after 10 min:

          root@jenkins:/var/log/jenkins# systemctl status jenkins
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)
     Active: activating (start) since Fri 2024-04-05 05:55:07 UTC; 10min ago
   Main PID: 1886 (java)
      Tasks: 95 (limit: 38470)
     Memory: 267.0M (peak: 286.2M)
        CPU: 13.224s
     CGroup: /system.slice/jenkins.service
             └─1886 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --logfile=/var/log/jenkins/jenkins.log --httpPort=8080Apr 05 05:55:07 jenkins systemd[1]: Starting jenkins.service - Jenkins Continuous Integration Server...
Apr 05 05:55:07 jenkins jenkins[1886]: Running from: /usr/share/java/jenkins.war
root@jenkins:/var/log/jenkins# jstack 1886 > jstack_10min.txt

          collected:
          jjstack_10min.txt
          jenkins.logfrom var/log
          jenkins.serviceactual systemd configuration

          shall I increase "JENKINS_DEBUG_LEVEL=5" to 10 or higher to get debug logs?

          L added a comment - Basil thanks for the quick reply! Just increased to TimeoutStartSec=900. also just modified to logging into var/log Let me share the results after 10 min: root@jenkins:/ var /log/jenkins# systemctl status jenkins ● jenkins.service - Jenkins Continuous Integration Server      Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)      Active: activating (start) since Fri 2024-04-05 05:55:07 UTC; 10min ago    Main PID: 1886 (java)       Tasks: 95 (limit: 38470)      Memory: 267.0M (peak: 286.2M)         CPU: 13.224s      CGroup: /system.slice/jenkins.service              └─1886 /usr/bin/java -Djava.awt.headless= true -jar /usr/share/java/jenkins.war --webroot=/ var /cache/jenkins/war --logfile=/ var /log/jenkins/jenkins.log --httpPort=8080Apr 05 05:55:07 jenkins systemd[1]: Starting jenkins.service - Jenkins Continuous Integration Server... Apr 05 05:55:07 jenkins jenkins[1886]: Running from: /usr/share/java/jenkins.war root@jenkins:/ var /log/jenkins# jstack 1886 > jstack_10min.txt collected: j jstack_10min.txt jenkins.log from var/log jenkins.service actual systemd configuration shall I increase "JENKINS_DEBUG_LEVEL=5" to 10 or higher to get debug logs?

          Basil Crow added a comment -

          Looks like it's stuck on hudson.model.DownloadService.loadJSON(DownloadService.java:122) presumably trying to fetch https://updates.jenkins.io/update-center.json. Is your server unable to reach the Update Center or unable to download a 3 MiB JSON file?

          Basil Crow added a comment - Looks like it's stuck on hudson.model.DownloadService.loadJSON(DownloadService.java:122) presumably trying to fetch https://updates.jenkins.io/update-center.json . Is your server unable to reach the Update Center or unable to download a 3 MiB JSON file?

          L added a comment - 
          root@jenkins:/var/lib/jenkins/plugins# wget https://updates.jenkins.io/update-center.json
--2024-04-05 06:45:15--  https://updates.jenkins.io/update-center.json
Connecting to 87.254.212.120:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://updates.jenkins.io/current/update-center.json [following]
--2024-04-05 06:45:16--  https://updates.jenkins.io/current/update-center.json
Reusing existing connection to updates.jenkins.io:443.
Proxy request sent, awaiting response... 200 OK
Length: 2959980 (2.8M) [application/json]
Saving to: ‘update-center.json’update-center.json                              100%[====================================================================================================>]   2.82M  2.47MB/s    in 1.1s2024-04-05 06:45:17 (2.47 MB/s) - ‘update-center.json’ saved [2959980/2959980]

          I don't really understand.. Let me run again

          L added a comment - root@jenkins:/ var /lib/jenkins/plugins# wget https: //updates.jenkins.io/update-center.json --2024-04-05 06:45:15--  https: //updates.jenkins.io/update-center.json Connecting to 87.254.212.120:8080... connected. Proxy request sent, awaiting response... 301 Moved Permanently Location: https: //updates.jenkins.io/current/update-center.json [following] --2024-04-05 06:45:16--  https: //updates.jenkins.io/current/update-center.json Reusing existing connection to updates.jenkins.io:443. Proxy request sent, awaiting response... 200 OK Length: 2959980 (2.8M) [application/json] Saving to: ‘update-center.json’update-center.json                              100%[====================================================================================================>]   2.82M  2.47MB/s    in 1.1s2024-04-05 06:45:17 (2.47 MB/s) - ‘update-center.json’ saved [2959980/2959980] I don't really understand.. Let me run again

          L added a comment - - edited

          Think so the connection is ok, I need proxy to reach the external network - maybe it was failed last time
          jstack dump seems different now
          jenkins_v2.log

          jstack_11min.txt

          root@jenkins:/var/log/jenkins# systemctl status jenkins
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)
     Active: activating (start) since Fri 2024-04-05 06:45:33 UTC; 11min ago
   Main PID: 7299 (java)
      Tasks: 95 (limit: 38470)
     Memory: 279.9M (peak: 286.0M)
        CPU: 14.422s
     CGroup: /system.slice/jenkins.service
             └─7299 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --logfile=/var/log/jenkins/jenkins.log --httpPort=8080 --debug=9Apr 05 06:45:33 jenkins systemd[1]: Starting jenkins.service - Jenkins Continuous Integration Server...
Apr 05 06:45:33 jenkins jenkins[7299]: Running from: /usr/share/java/jenkins.war
root@jenkins:/var/log/jenkins# jstack 7299 > jstack_11min.txt

           

          L added a comment - - edited Think so the connection is ok, I need proxy to reach the external network - maybe it was failed last time jstack dump seems different now jenkins_v2.log jstack_11min.txt root@jenkins:/ var /log/jenkins# systemctl status jenkins ● jenkins.service - Jenkins Continuous Integration Server      Loaded: loaded (/usr/lib/systemd/system/jenkins.service; enabled; preset: enabled)      Active: activating (start) since Fri 2024-04-05 06:45:33 UTC; 11min ago    Main PID: 7299 (java)       Tasks: 95 (limit: 38470)      Memory: 279.9M (peak: 286.0M)         CPU: 14.422s      CGroup: /system.slice/jenkins.service              └─7299 /usr/bin/java -Djava.awt.headless= true -jar /usr/share/java/jenkins.war --webroot=/ var /cache/jenkins/war --logfile=/ var /log/jenkins/jenkins.log --httpPort=8080 --debug=9Apr 05 06:45:33 jenkins systemd[1]: Starting jenkins.service - Jenkins Continuous Integration Server... Apr 05 06:45:33 jenkins jenkins[7299]: Running from: /usr/share/java/jenkins.war root@jenkins:/ var /log/jenkins# jstack 7299 > jstack_11min.txt  

          Basil Crow added a comment -

          The stack trace is the same. You somehow need to configure Java and/or Jenkins to use your proxy. I don't think there is a Jenkins core bug here.

          Basil Crow added a comment - The stack trace is the same. You somehow need to configure Java and/or Jenkins to use your proxy. I don't think there is a Jenkins core bug here.

          L added a comment -

          thank you for the investigation!

          L added a comment - thank you for the investigation!

          L added a comment - - edited

          Basil, I searched over the internet how to setup proxy for Java or Jenkins, but I found only that if I set global proxy to use like this:

          export https_proxy=http://87.254.212.120:8080

          System will use it. (And this was already done..) So don't really know what to do with this situation.

          L added a comment - - edited Basil, I searched over the internet how to setup proxy for Java or Jenkins, but I found only that if I set global proxy to use like this: export https_proxy=http: //87.254.212.120:8080 System will use it. (And this was already done..) So don't really know what to do with this situation.

          Basil Crow added a comment -

          This is normally configured via the GUI in Manage Jenkins > System, but if you can't complete the setup wizard you can try manually creating a proxy.xml file in $JENKINS_HOME:

          <?xml version='1.1' encoding='UTF-8'?>
<proxy>
  <name>proxy.example.com</name>
  <port>8080</port>
  <userName></userName>
  <noProxyHost></noProxyHost>
  <secretPassword></secretPassword>
  <testUrl></testUrl>
</proxy>

          Basil Crow added a comment - This is normally configured via the GUI in Manage Jenkins > System, but if you can't complete the setup wizard you can try manually creating a proxy.xml file in $JENKINS_HOME : <?xml version='1.1' encoding='UTF-8'?> <proxy> <name>proxy.example.com</name> <port>8080</port> <userName></userName> <noProxyHost></noProxyHost> <secretPassword></secretPassword> <testUrl></testUrl> </proxy>

          L added a comment -

          found a way to start it

          /usr/bin/java
-Djava.awt.headless=true -Dhttp.proxyHost=87.254.212.120 -Dhttp.proxyPort=8080
-jar /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war
--logfile=/var/log/jenkins/jenkins.log --httpPort=8080 --debug=9 > /dev/null
2>&1 & echo $!

          but there is something I could not understand.. the same command not working if the daemon starts it.
          I 've modified the daemon (cat /usr/lib/systemd/system/jenkins.service) 

          Environment="JAVA_OPTS=-Djava.awt.headless=true -Dhttp.proxyHost=87.254.212.120 -Dhttp.proxyPort=8080"

          in ps aux its clearly visible the daemon executes the same, but jenkins does not start properly.
          any idea why is it not working from via the daemon?

          L added a comment - found a way to start it /usr/bin/java -Djava.awt.headless= true -Dhttp.proxyHost=87.254.212.120 -Dhttp.proxyPort=8080 -jar /usr/share/java/jenkins.war --webroot=/ var /cache/jenkins/war --logfile=/ var /log/jenkins/jenkins.log --httpPort=8080 --debug=9 > /dev/ null 2>&1 & echo $! but there is something I could not understand.. the same command not working if the daemon starts it. I 've modified the daemon (cat /usr/lib/systemd/system/jenkins.service)  Environment= "JAVA_OPTS=-Djava.awt.headless= true -Dhttp.proxyHost=87.254.212.120 -Dhttp.proxyPort=8080" in ps aux its clearly visible the daemon executes the same, but jenkins does not start properly. any idea why is it not working from via the daemon?

          Basil Crow added a comment -

          Not sure, maybe some quoting issue.

          Basil Crow added a comment - Not sure, maybe some quoting issue.

            Unassigned Unassigned
            milkuci L
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: