Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65809

Jenkins fails to restart after plugin updates on Debian 11 (bullseye)

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • 2.335, 2.332.1

      Jenkins fails to restart after plugin updates, when installed via Debian package

      To reproduce:

      1. Install Jenkins from https://pkg.jenkins.io/debian-stable 
      2. Install some plugin updates via the Jenkins web UI, and tell Jenkins to restart when done
      3. Observe that the "Please wait while Jenkins is restarting ..." page remains forever

      Looking into the logs, there is:

      WARNING jenkins.model.Jenkins$20#run: Failed to restart Jenkins
      java.io.IOException: Failed to exec 'jenkins: /bin/java' No such file or directory
              at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:87)
              at jenkins.model.Jenkins$20.run(Jenkins.java:4397) 

      Naturally, trying to execute a command named 'jenkins: /bin/java' failed. But why did it try?

      Looking at the running Jenkins process, its /proc/<PID>/cmdline value has been updated to have 'jenkins: /bin/java'.

      It is being run under the /usr/bin/daemon process manager, which performs this name update to make the identity of the service process clearer in process listings.

       

      The Jenkins Debian packages should not install Jenkins in a way which defeats its own auto-restart functionality.

          [JENKINS-65809] Jenkins fails to restart after plugin updates on Debian 11 (bullseye)

          Mark Waite added a comment -

          I can't duplicate the problem that you're describing. The steps that I took while trying to duplicate the problem were:

          1. Remove Jenkins 2.299 from my Debian 10 (buster) and confirm that all Jenkins files are removed
          2. Install Jenkins 2.300 on my Debian 10 (buster) using the Debian package
          3. Provide the temporary password, install suggested plugins, and confirm that Jenkins starts
          4. Install blue ocean plugin (and its dependencies) from plugin manager
          5. Restart Jenkins and confirm that the restart is successful
          6. Upload elastic axis plugin 1.3 (intentionally outdated plugin) using the plugin manager "Advanced" tab and restart Jenkins
          7. Confirm that an upgrade from elastic axis plugin is available
          8. Install the plugin upgrade and restart Jenkins
          9. Confirm that Jenkins restarts successfully

          You might check to see if you have inadvertently set JAVA_HOME to 'jenkins: /bin/java' by mistake. You might need to explore other configuration issues on that computer.

          Mark Waite added a comment - I can't duplicate the problem that you're describing. The steps that I took while trying to duplicate the problem were: Remove Jenkins 2.299 from my Debian 10 (buster) and confirm that all Jenkins files are removed Install Jenkins 2.300 on my Debian 10 (buster) using the Debian package Provide the temporary password, install suggested plugins, and confirm that Jenkins starts Install blue ocean plugin (and its dependencies) from plugin manager Restart Jenkins and confirm that the restart is successful Upload elastic axis plugin 1.3 (intentionally outdated plugin) using the plugin manager "Advanced" tab and restart Jenkins Confirm that an upgrade from elastic axis plugin is available Install the plugin upgrade and restart Jenkins Confirm that Jenkins restarts successfully You might check to see if you have inadvertently set JAVA_HOME to 'jenkins: /bin/java' by mistake. You might need to explore other configuration issues on that computer.

          Simon Stiil added a comment -

          Get the same issue.

          Debian 11 fresh install from netboot image. (VM running in proxmox)

          openjdk-11-jre openjdk-11-jdk for java installation.

          Which creates symlinks for both /usr/bin/java and /bin/java -> /etc/alternatives/java -> /usr/lib/jvm/java-11-openjdk-amd64/bin/java

          installing the debian-stable jenkins package. (2.289.2)

          starting and stopping using /etc/init.d/jenkins works fine no issues.

          Restarting from jenkins during plugin installation makes jenkins hang with waiting for jenkins to start in web intarface. And log says:

          2021-07-15 10:17:00.307+0000 [id=158] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped

          2021-07-15 10:17:00.926+0000 [id=158] WARNING jenkins.model.Jenkins$20#run: Failed to restart Jenkins

          java.io.IOException: Failed to exec 'jenkins: /bin/java' No such file or directory

              at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:87)

              at jenkins.model.Jenkins$20.run(Jenkins.java:4397)

           

           

          Simon Stiil added a comment - Get the same issue. Debian 11 fresh install from netboot image. (VM running in proxmox) openjdk-11-jre openjdk-11-jdk for java installation. Which creates symlinks for both /usr/bin/java and /bin/java -> /etc/alternatives/java -> /usr/lib/jvm/java-11-openjdk-amd64/bin/java installing the debian-stable jenkins package. (2.289.2) starting and stopping using /etc/init.d/jenkins works fine no issues. Restarting from jenkins during plugin installation makes jenkins hang with waiting for jenkins to start in web intarface. And log says: 2021-07-15 10:17:00.307+0000 [id=158] INFO jenkins.model.Jenkins#cleanUp: Jenkins stopped 2021-07-15 10:17:00.926+0000 [id=158] WARNING jenkins.model.Jenkins$20#run: Failed to restart Jenkins java.io.IOException: Failed to exec 'jenkins: /bin/java' No such file or directory     at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:87)     at jenkins.model.Jenkins$20.run(Jenkins.java:4397)    

          Max added a comment -

          I see why different people are getting different results - the change to include the service name in the ps listing is a new "feature" in daemon 0.7, shipped in Debian 11 (bullseye).

          This makes the existing Jenkins Debian package fail to handle restarts properly on Debian 11, but not 10.

           

          One fairly simple fix would be to stop using this daemon package, and just use systemd instead.

           

          I created /etc/systemd/system/jenkins.service with contents

          [Unit]
          Description=Jenkins
          Wants=network-online.target
          After=network-online.target
          
          [Service]
          Type=exec
          Environment=JENKINS_HOME=/var/lib/jenkins
          ExecStart=/usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war \
            --webroot=/var/cache/jenkins/war \
            --httpPort=8080 \
            --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger \
            --simpleAccessLogger.format=combined \
            --simpleAccessLogger.file=/var/log/jenkins/access_log
          User=jenkins
          
          [Install]
          WantedBy=multi-user.target

          thus bypassing the /etc/init.d/jenkins and /etc/default/jenkins shipped in the package entirely, and this has allowed the automatic restart to work for me.

          Max added a comment - I see why different people are getting different results - the change to include the service name in the ps listing is a new "feature" in daemon 0.7, shipped in Debian 11 (bullseye). This makes the existing Jenkins Debian package fail to handle restarts properly on Debian 11, but not 10.   One fairly simple fix would be to stop using this daemon package, and just use systemd instead.   I created /etc/systemd/system/jenkins.service with contents [Unit] Description=Jenkins Wants=network-online.target After=network-online.target [Service] Type=exec Environment=JENKINS_HOME=/var/lib/jenkins ExecStart=/usr/bin/java -Djava.awt.headless=true -jar /usr/share/jenkins/jenkins.war \ --webroot=/var/cache/jenkins/war \ --httpPort=8080 \ --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger \ --simpleAccessLogger.format=combined \ --simpleAccessLogger.file=/var/log/jenkins/access_log User=jenkins [Install] WantedBy=multi-user.target thus bypassing the /etc/init.d/jenkins and /etc/default/jenkins shipped in the package entirely, and this has allowed the automatic restart to work for me.

          Same behavior with a fresh install on Debian Bullseye, with openjdk-11-jdk package and jenkins package from LTS repository.

          Jenkins fails to restart either from https://$JENKINS_URL/restart or from Jenkins.instance.safeRestart().

          The change comes indeed from daemon 0.7, daemon.c@4343, which updates the process command line.

          if (asprintf(&g.cmd[0], "%s: %s", g.name, g.cmdpath) == -1)
          

          I've set a monitoring agent to restart completly the instance whenever a programmatic restart was triggered, but this is obviously suboptimal.

          Thomas Duboucher added a comment - Same behavior with a fresh install on Debian Bullseye, with openjdk-11-jdk package and jenkins package from LTS repository. Jenkins fails to restart either from https://$JENKINS_URL/restart or from Jenkins.instance.safeRestart() . The change comes indeed from daemon 0.7, daemon.c@4343, which updates the process command line. if (asprintf(&g.cmd[0], "%s: %s" , g.name, g.cmdpath) == -1) I've set a monitoring agent to restart completly the instance whenever a programmatic restart was triggered, but this is obviously suboptimal.

          Tom added a comment - - edited

          I'm seeing the same problem when running Jenkins on Devuan 4 (compatible with Debian 11) with sysvinit.

          2022-01-04 22:36:55.473+0000 [id=246]   WARNING jenkins.model.Jenkins$20#run: Failed to restart Jenkins
          java.io.IOException: Failed to exec 'jenkins: /usr/bin/java' No such file or directory
                  at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:86)
                  at jenkins.model.Jenkins$20.run(Jenkins.java:4513)
          

          Devuan does not use SystemD so please do not change to a solution relying only on SystemD as it will prevent Jenkins from running on Devuan.

          Tom added a comment - - edited I'm seeing the same problem when running Jenkins on Devuan 4 (compatible with Debian 11) with sysvinit. 2022-01-04 22:36:55.473+0000 [id=246] WARNING jenkins.model.Jenkins$20#run: Failed to restart Jenkins java.io.IOException: Failed to exec 'jenkins: /usr/bin/java' No such file or directory at hudson.lifecycle.UnixLifecycle.restart(UnixLifecycle.java:86) at jenkins.model.Jenkins$20.run(Jenkins.java:4513) Devuan does not use SystemD so please do not change to a solution relying only on SystemD as it will prevent Jenkins from running on Devuan.

          Karoly Balogh added a comment -

          I've ran into this recently on one of the Jenkins setups I maintain, after upgrading to Debian 11.

          The issue indeed comes from daemon 0.7, a package which was updated after 10+ years... It's even noted in its Changelog:

          `Insert daemon name before client's argv[0] so it appears in ps output (for marko.asplund at gmail.com)`

          I think this change should really be reverted in daemon, overwriting the client's argv[0] is a really dumb idea IMO, but at least it should be made optional while keeping other functionality... But anyway, more relevant to Jenkins: A potential workaround can be removing the `--name` argument from `/etc/init.d/jenkins`, in the `DAEMON_ARGS=` line, so it reads:

          `DAEMON_ARGS="--inherit --env=JENKINS_HOME=$JENKINS_HOME --output=$JENKINS_LOG --pidfile=$PIDFILE"`

          instead of the original

          `DAEMON_ARGS="--name=$NAME --inherit --env=JENKINS_HOME=$JENKINS_HOME --output=$JENKINS_LOG --pidfile=$PIDFILE"`

          This obviously isn't ideal, due to `–name` is also used to guarantee a single named instance, but at least deals with the restart part of the problem, at least on my system.

          Karoly Balogh added a comment - I've ran into this recently on one of the Jenkins setups I maintain, after upgrading to Debian 11. The issue indeed comes from daemon 0.7, a package which was updated after 10+ years... It's even noted in its Changelog: `Insert daemon name before client's argv [0] so it appears in ps output (for marko.asplund at gmail.com)` I think this change should really be reverted in daemon, overwriting the client's argv [0] is a really dumb idea IMO, but at least it should be made optional while keeping other functionality... But anyway, more relevant to Jenkins: A potential workaround can be removing the `--name` argument from `/etc/init.d/jenkins`, in the `DAEMON_ARGS=` line, so it reads: `DAEMON_ARGS="--inherit --env=JENKINS_HOME=$JENKINS_HOME --output=$JENKINS_LOG --pidfile=$PIDFILE"` instead of the original `DAEMON_ARGS="--name=$NAME --inherit --env=JENKINS_HOME=$JENKINS_HOME --output=$JENKINS_LOG --pidfile=$PIDFILE"` This obviously isn't ideal, due to `–name` is also used to guarantee a single named instance, but at least deals with the restart part of the problem, at least on my system.

          Mark Waite added a comment -

          chainq the Jenkins 2.333 weekly release on Debian no longer requires the daemon package. It uses the start-stop-daemon facility instead of using the daemon package.

          See PR-261 for the details of the change.

          Mark Waite added a comment - chainq the Jenkins 2.333 weekly release on Debian no longer requires the daemon package. It uses the start-stop-daemon facility instead of using the daemon package. See PR-261 for the details of the change.

          Karoly Balogh added a comment -

          markewaite Great, thanks!. So when that goes LTS, the problem will be solved, and I can probably live with the workaround for now. (Or I could just import the new init script from that PR too meanwhile.) Anyway, great news.

          Karoly Balogh added a comment - markewaite Great, thanks!. So when that goes LTS, the problem will be solved, and I can probably live with the workaround for now. (Or I could just import the new init script from that PR too meanwhile.) Anyway, great news.

          Mark Waite added a comment -

          Flagged as a possible LTS candidate for Jenkins 2.232.1 but will need further testing and "soak time" before the changes are applied to the stable-2.332 branch of the packaging repository.

          Mark Waite added a comment - Flagged as a possible LTS candidate for Jenkins 2.232.1 but will need further testing and "soak time" before the changes are applied to the stable-2.332 branch of the packaging repository.

            Unassigned Unassigned
            maxb Max
            Votes:
            9 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: