Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67606

Jenkins start issue in CentOS-7 host since version jenkins-2.312.war

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • packaging
    • None
    • CentOS Linux release 7.9.2009

      Recently we tried to upgrade our jenkins master from jenkins-2.287.war to jenkins-2.321.war. We did the same jenkins upgrade and its plugins upgrade in one Mac host. I was successfully.

      But in one CentOS-7 VM host, it failed with below error message.

      jenkins.service - LSB: Jenkins Automation Server
        Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
        Active: failed (Result: timeout) since Wed 2022-01-12 02:30:49 PST; 33min ago
         Docs: man:systemd-sysv-generator(8)
       Process: 31106 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)
       Process: 32363 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)
        Tasks: 264
        CGroup: /system.slice/jenkins.service
            ├─32370 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkin...
            ├─32371 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/...
            └─32372 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/je...Jan 12 02:25:49 jenkins-lnx.filemaker.com systemd[1]: Starting LSB: Jenkins Automation Server...
      Jan 12 02:25:49 jenkins-lnx.filemaker.com runuser[32370]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
      Jan 12 02:30:49 jenkins-lnx.filemaker.com systemd[1]jenkins.service start operation timed out. Terminating.
      Jan 12 02:30:49 jenkins-lnx.filemaker.com systemd[1]Failed to start LSB: Jenkins Automation Server.
      Jan 12 02:30:49 jenkins-lnx.filemaker.com systemd[1]Unit jenkins.service entered failed state.

      But we still could login into jenkins and ran jobs.

      If we rolled back jenkins version to jenkins-2.311.war, 'sudo systemctl start jenkins' worked well.

      Please let me know if you need more information to fix this issue.

        1. jenkins.log
          165 kB
        2. jenkins.log2
          51 kB
        3. jenkins.log3
          43 kB
        4. Screen Shot 2022-01-24 at 12.50.07 PM.png
          Screen Shot 2022-01-24 at 12.50.07 PM.png
          741 kB

          [JENKINS-67606] Jenkins start issue in CentOS-7 host since version jenkins-2.312.war

          Basil Crow added a comment -

          The relevant error message would be in /var/log/jenkins/jenkins.log.

          Basil Crow added a comment - The relevant error message would be in /var/log/jenkins/jenkins.log .

          Jianhui He added a comment -

          From my jenkins log, it complained "

          2022-01-24 20:44:44.120+0000 [id=34]    WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues

          java.lang.RuntimeException: unsupported Java version: 11

          "

          Even after I restarted jenkins, from jenkins plugins manager, it sill showed ruby-runtime plugin "Failed to initialize".

           

          Jianhui He added a comment - From my jenkins log, it complained " 2022-01-24 20:44:44.120+0000 [id=34]     WARNING jenkins.model.Jenkins$5#runTask: Loading plugin ruby-runtime v0.12 (ruby-runtime) failed perhaps due to plugin dependency issues java.lang.RuntimeException: unsupported Java version: 11 " Even after I restarted jenkins, from jenkins plugins manager, it sill showed ruby-runtime plugin "Failed to initialize".  

          Mark Waite added a comment -

          The ruby runtime plugin does not support Java 11. You'll either need to remove that plugin or configure your system to run with Java 8.

          See https://www.jenkins.io/blog/2021/12/22/deprecated-ruby-runtime/ for the announced removal of all plugins that require the ruby runtime plugin. Jenkins plugins based on the JRuby runtime will no longer distributed by the Jenkins project.

          Mark Waite added a comment - The ruby runtime plugin does not support Java 11. You'll either need to remove that plugin or configure your system to run with Java 8. See https://www.jenkins.io/blog/2021/12/22/deprecated-ruby-runtime/ for the announced removal of all plugins that require the ruby runtime plugin. Jenkins plugins based on the JRuby runtime will no longer distributed by the Jenkins project.

          Jianhui He added a comment -

          Originally our jenkins host ran java-8. When it ran into this jenkins start up issue, we upgraded its java version to java-11. Now I rolled back its java to java8. Now it didn't complain rub-runtime plugin issue but it still ran into the same startup issue. I will upload its full jenkins log soon

          Jianhui He added a comment - Originally our jenkins host ran java-8. When it ran into this jenkins start up issue, we upgraded its java version to java-11. Now I rolled back its java to java8. Now it didn't complain rub-runtime plugin issue but it still ran into the same startup issue. I will upload its full jenkins log soon

          Jianhui He added a comment -

          Now it complained below error message

          2022-01-24 21:25:54.732+0000 [id=485]   WARNING h.i.i.InstallUncaughtExceptionHandler#handleException

          java.io.IOException: Close SendCallback@3b0c0ea2[PROCESSING][i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6f3766c7] in state PROCESSING

                  at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:428)

                  at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:519)

                  at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:378)

                  at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)

                  at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1104)

          Caused: javax.servlet.ServletException

          Jianhui He added a comment - Now it complained below error message 2022-01-24 21:25:54.732+0000 [id=485]   WARNING h.i.i.InstallUncaughtExceptionHandler#handleException java.io.IOException: Close SendCallback@3b0c0ea2 [PROCESSING] [i=null,cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6f3766c7] in state PROCESSING         at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:428)         at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:519)         at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:378)         at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345)         at org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint.run(ManagedSelector.java:1104) Caused: javax.servlet.ServletException

          Jianhui He added a comment -

          How did you fix it? Or how should I fix it?

          Jianhui He added a comment - How did you fix it? Or how should I fix it?

          Jianhui He added a comment -

          I tried your latest version jenkins-2.335.war but it still got the same issue.

          when running sudo systemctl start jenkins, it got below process running

          [builder@jenkins-lnx ~]$ ps -ef |grep java

          root     18039 18034  0 09:55 ?        00:00:00 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword='SCMLAB2021' --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

          jenkins  18040 18039  0 09:55 ?        00:00:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword='SCMLAB2021' --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

          jenkins  18041 18040 99 09:55 ?        00:06:14 /etc/alternatives/java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword=SCMLAB2021 --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

          builder  19037 18127  0 09:57 pts/1    00:00:00 grep --color=auto java

          when running sudo systemctl status jenkins, it got below message

          [builder@jenkins-lnx ~]$ sudo systemctl status jenkins

          ● jenkins.service - LSB: Jenkins Automation Server

             Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)

            Drop-In: /etc/systemd/system/jenkins.service.d

                     └─startup-timeout.conf

             Active: activating (start) since Thu 2022-02-17 09:55:15 PST; 4min 8s ago

               Docs: man:systemd-sysv-generator(8)

            Process: 17975 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)

            Control: 18034 (jenkins)

              Tasks: 198

             CGroup: /system.slice/jenkins.service

                     ├─18034 /bin/sh /etc/rc.d/init.d/jenkins start

                     ├─18039 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenki...

                     ├─18040 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/lo...

                     └─18041 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cach...

           

          Feb 17 09:55:15 jenkins-lnx.filemaker.com systemd[1]: Starting LSB: Jenkins Automation Server...

          Feb 17 09:55:15 jenkins-lnx.filemaker.com runuser[18039]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)

          But cmd "sudo systemctl start jenkins" was still stuck in the terminal. After several minutes timeout, it finished from terminal and got below message

          Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details.

          If run sudo systemctl status jenkins.service, it got below message

          [builder@jenkins-lnx ~]$ sudo systemctl status jenkins

          jenkins.service - LSB: Jenkins Automation Server

             Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)

            Drop-In: /etc/systemd/system/jenkins.service.d

                     └─startup-timeout.conf

             Active: failed (Result: timeout) since Thu 2022-02-17 10:02:15 PST; 3min 26s ago

               Docs: man:systemd-sysv-generator(8)

            Process: 17975 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)

            Process: 18034 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)

              Tasks: 189

             CGroup: /system.slice/jenkins.service

                     ├─18039 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenki...

                     ├─18040 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/lo...

                     └─18041 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cach...

           

          Feb 17 09:55:15 jenkins-lnx.filemaker.com systemd[1]: Starting LSB: Jenkins Automation Server...

          Feb 17 09:55:15 jenkins-lnx.filemaker.com runuser[18039]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)

          Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd[1]: jenkins.service start operation timed out. Terminating.

          Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd[1]: Failed to start LSB: Jenkins Automation Server.

          Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd[1]: Unit jenkins.service entered failed state.

          Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd[1]: jenkins.service failed.

           

           

          Jianhui He added a comment - I tried your latest version jenkins-2.335.war but it still got the same issue. when running sudo systemctl start jenkins, it got below process running [builder@jenkins-lnx ~] $ ps -ef |grep java root     18039 18034  0 09:55 ?        00:00:00 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/ java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword='SCMLAB2021' --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 jenkins  18040 18039  0 09:55 ?        00:00:00 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/ java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword='SCMLAB2021' --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 jenkins  18041 18040 99 09:55 ?        00:06:14 /etc/alternatives/ java -D*java*.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=-1 --httpsPort=8443 --httpsKeyStore=/usr/lib/jenkins/Certification/jenkins-lnx.jks --httpsKeyStorePassword=SCMLAB2021 --httpsListenAddress=0.0.0.0 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 builder  19037 18127  0 09:57 pts/1    00:00:00 grep --color=auto java when running  sudo systemctl status jenkins, it got below message [builder@jenkins-lnx ~] $ sudo systemctl status jenkins ● jenkins.service - LSB: Jenkins Automation Server    Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)   Drop-In: /etc/systemd/system/jenkins.service.d            └─startup-timeout.conf    Active: activating (start) since Thu 2022-02-17 09:55:15 PST; 4min 8s ago      Docs: man:systemd-sysv-generator(8)   Process: 17975 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)   Control: 18034 (jenkins)     Tasks: 198    CGroup: /system.slice/jenkins.service            ├─18034 /bin/sh /etc/rc.d/init.d/jenkins start            ├─18039 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenki...            ├─18040 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/lo...            └─18041 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cach...   Feb 17 09:55:15 jenkins-lnx.filemaker.com systemd [1] : Starting LSB: Jenkins Automation Server... Feb 17 09:55:15 jenkins-lnx.filemaker.com runuser [18039] : pam_unix(runuser:session): session opened for user jenkins by (uid=0) But cmd "sudo systemctl start jenkins" was still stuck in the terminal. After several minutes timeout, it finished from terminal and got below message Job for jenkins.service failed because a timeout was exceeded. See "systemctl status jenkins.service" and "journalctl -xe" for details. If run sudo systemctl status jenkins.service, it got below message [builder@jenkins-lnx ~] $ sudo systemctl status jenkins ● jenkins.service - LSB: Jenkins Automation Server    Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)   Drop-In: /etc/systemd/system/jenkins.service.d            └─startup-timeout.conf    Active: failed (Result: timeout) since Thu 2022-02-17 10:02:15 PST; 3min 26s ago      Docs: man:systemd-sysv-generator(8)   Process: 17975 ExecStop=/etc/rc.d/init.d/jenkins stop (code=exited, status=0/SUCCESS)   Process: 18034 ExecStart=/etc/rc.d/init.d/jenkins start (code=killed, signal=TERM)     Tasks: 189    CGroup: /system.slice/jenkins.service            ├─18039 runuser -s /bin/bash jenkins -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenki...            ├─18040 bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/lo...            └─18041 /etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/home/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cach...   Feb 17 09:55:15 jenkins-lnx.filemaker.com systemd [1] : Starting LSB: Jenkins Automation Server... Feb 17 09:55:15 jenkins-lnx.filemaker.com runuser [18039] : pam_unix(runuser:session): session opened for user jenkins by (uid=0) Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd [1] : jenkins.service start operation timed out. Terminating. Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd [1] : Failed to start LSB: Jenkins Automation Server. Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd [1] : Unit jenkins.service entered failed state. Feb 17 10:02:15 jenkins-lnx.filemaker.com systemd [1] : jenkins.service failed.    

          Mark Waite added a comment -

          jianhui_he I think that you've misunderstood how to upgrade Jenkins that has been installed with the operating system rpm package. If Jenkins is installed with the operating system package manager, it must be upgraded with the operating system package manager. The `yum upgrade` command should be used to perform upgrades.

          When you replace the war file with a newer copy, you bypass all the code that the operating system package manager would have run to update your Jenkins installation from using System V init to using systemd.

          Mark Waite added a comment - jianhui_he I think that you've misunderstood how to upgrade Jenkins that has been installed with the operating system rpm package. If Jenkins is installed with the operating system package manager, it must be upgraded with the operating system package manager. The `yum upgrade` command should be used to perform upgrades. When you replace the war file with a newer copy, you bypass all the code that the operating system package manager would have run to update your Jenkins installation from using System V init to using systemd.

          Jianhui He added a comment -

          After running sudo yum install jenkins, our jenkins was successfully upgraded to 2.335.

          Jianhui He added a comment - After running sudo yum install jenkins, our jenkins was successfully upgraded to 2.335.

            Unassigned Unassigned
            jianhui_he Jianhui He
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: