Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67487

Start Jenkins command starts but does not return back to shell on Amazon Linux 2

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • packaging
    • None
    • Jenkins server 2.319.1
    • 2.333

      After upgrade to 2.319.1 (from 2.277.2), the command "service jenkins start" on the Linux console hangs (we waited for ~30 mins). It doesnt return back to shell. This currently prevents the ansible playbook from progressing since it gets stuck at this task waiting for a return to shell.

      previously, the 'service jenkins start' command should used to return to shell with 5 seconds.

      Eg:

      service jenkins start

      [ABC@ip-ABC]# service jenkins start
      Starting Jenkins
       
      Session terminated, killing shell... ...killed.
      [ABC@ ~]#
      

      Debugging the init.d script for jenkins, the daemon command which used to return back to the shell, doesnt do that anymore.

      echo -n "Starting Jenkins "
              daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" "$JAVA_CMD" $PARAMS > /dev/null
              RETVAL=$?
      

      Values of the variables:

      JENKINS_USER = jenkins

      JENKINS_PID_FILE = /var/run/jenkins.pid

      JAVA_CMD = /etc/alternatives/java

      PARAMS =  -Xmx3883m -Xms3883m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dhudson.model.DirectoryBrowserSupport.CSP="default-src 'self'; connect-src 'self' 'unsafe-inline' storybook.js.org; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self' 'unsafe-inline';" -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war and --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

       

      We see that while the service start command is stuck, it is able to start Jenkins successfully. 

      We confirmed:

      1. Jenkins process (ps output) is up.
      2. /var/log/jenkins/jenkins.log has "Jenkins is fully up and running"

          [JENKINS-67487] Start Jenkins command starts but does not return back to shell on Amazon Linux 2

          Abhinav Varshney created issue -
          Abhinav Varshney made changes -
          Description Original: After upgrade to 2.319.1 (from 2.277.2), the command "service jenkins start" on the Linux console hangs (we waited for ~30 mins). It doesnt return back to shell. This currently prevents the ansible playbook from progressing since it gets stuck at this task waiting for a return to shell.

          previously, the 'service jenkins start' command should used to return to shell with 5 seconds.

          Eg:

          service jenkins start
          {code:java}
          [ABC@ip-ABC]# service jenkins start
          Starting Jenkins
           
          Session terminated, killing shell... ...killed.
          [ABC@ ~]#
          {code}
          Debugging the init.d script for jenkins, the daemon command which used to return back to the shell, doesnt do that anymore.
          {code:java}
          echo -n "Starting Jenkins "
                  daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" "$JAVA_CMD" $PARAMS > /dev/null
                  RETVAL=$?
          {code}

          Values of the variables:

          JENKINS_USER = jenkins

          JENKINS_PID_FILE = /var/run/jenkins.pid

          JAVA_CMD = /etc/alternatives/java

          PARAMS =  -Xmx3883m -Xms3883m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dhudson.model.DirectoryBrowserSupport.CSP="default-src 'self'; connect-src 'self' 'unsafe-inline' storybook.js.org; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self' 'unsafe-inline';" -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war and --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

           

           
          New: After upgrade to 2.319.1 (from 2.277.2), the command "service jenkins start" on the Linux console hangs (we waited for ~30 mins). It doesnt return back to shell. This currently prevents the ansible playbook from progressing since it gets stuck at this task waiting for a return to shell.

          previously, the 'service jenkins start' command should used to return to shell with 5 seconds.

          Eg:

          service jenkins start
          {code:java}
          [ABC@ip-ABC]# service jenkins start
          Starting Jenkins
           
          Session terminated, killing shell... ...killed.
          [ABC@ ~]#
          {code}
          Debugging the init.d script for jenkins, the daemon command which used to return back to the shell, doesnt do that anymore.
          {code:java}
          echo -n "Starting Jenkins "
                  daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" "$JAVA_CMD" $PARAMS > /dev/null
                  RETVAL=$?
          {code}
          Values of the variables:

          JENKINS_USER = jenkins

          JENKINS_PID_FILE = /var/run/jenkins.pid

          JAVA_CMD = /etc/alternatives/java

          PARAMS =  -Xmx3883m -Xms3883m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dhudson.model.DirectoryBrowserSupport.CSP="default-src 'self'; connect-src 'self' 'unsafe-inline' storybook.js.org; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self' 'unsafe-inline';" -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war and --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

           

          We see that while the service start command is stuck, it is able to start Jenkins successfully. 

          We confirmed:
           # Jenkins process (ps output) is up.
           # /var/log/jenkins/jenkins.log has "Jenkins is fully up and running"
          Mark Waite made changes -
          Component/s New: packaging [ 20120 ]
          Component/s Original: command-launcher-plugin [ 23169 ]

          Mark Waite added a comment - - edited

          I've checked with Debian 10 (buster) and Debian 11 (bullseye) with Jenkins 2.319.1 and cannot duplicate the problem.

          What operating system are you using?

          Is it running the most recent patches?

          How did you install Jenkins?

          What is the output of /etc/alternatives/java -version?

          Mark Waite added a comment - - edited I've checked with Debian 10 (buster) and Debian 11 (bullseye) with Jenkins 2.319.1 and cannot duplicate the problem. What operating system are you using? Is it running the most recent patches? How did you install Jenkins? What is the output of /etc/alternatives/java -version ?

          Hi Mark,

          Thanks for quick response. We use Amzn Linux1 currently. 

          we installed jenkins using yum , version 2.277.3 a while ago and then upgraded to 2.319.1 via the UI upgrade option in Dec'21.

          We use JDK1.8

          [ABC@ABC ansible]# /etc/alternatives/java -version
          openjdk version "1.8.0_312"
          OpenJDK Runtime Environment (build 1.8.0_312-b07)
          OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
          [root@ip-10-224-53-73 ansible]#
          

          We have other jenkins instances that run on the same underlying OS + installed/upgraded Jenkins the same way. We are seeing this issue only with instances on version 2.319.1 Jenkins. 

           

          Abhinav Varshney added a comment - Hi Mark, Thanks for quick response. We use Amzn Linux1 currently.  we installed jenkins using yum , version 2.277.3 a while ago and then upgraded to 2.319.1 via the UI upgrade option in Dec'21. We use JDK1.8 [ABC@ABC ansible]# /etc/alternatives/java -version openjdk version "1.8.0_312" OpenJDK Runtime Environment (build 1.8.0_312-b07) OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode) [root@ip-10-224-53-73 ansible]# We have other jenkins instances that run on the same underlying OS + installed/upgraded Jenkins the same way. We are seeing this issue only with instances on version 2.319.1 Jenkins.   

          Mark Waite added a comment -

          I don't test with Amazon Linux 1. I don't know anyone else that tests with Amazon Linux 1. Amazon states on their web site that Amazon Linux AMI (Amazon Linux 1) entered maintenance mode in Dec 2020.

          I don't plan to investigate further on Amazon Linux 1. If you see a similar failure on Amazon Linux 2, that would be more likely to be investigated.

          Mark Waite added a comment - I don't test with Amazon Linux 1. I don't know anyone else that tests with Amazon Linux 1. Amazon states on their web site that Amazon Linux AMI (Amazon Linux 1) entered maintenance mode in Dec 2020. I don't plan to investigate further on Amazon Linux 1. If you see a similar failure on Amazon Linux 2, that would be more likely to be investigated.
          Mark Waite made changes -
          Summary Original: Start Jenkins command starts jenkins successfully but does not return back to shell New: Start Jenkins command starts jenkins successfully but does not return back to shell on Amazon Linux 1
          Mark Waite made changes -
          Summary Original: Start Jenkins command starts jenkins successfully but does not return back to shell on Amazon Linux 1 New: Start Jenkins command starts but does not return back to shell on Amazon Linux 1

          I understand. Is "service jenkins start" still the recommended way of starting up Jenkins on AMZN Linux 2? 

          Abhinav Varshney added a comment - I understand. Is "service jenkins start" still the recommended way of starting up Jenkins on AMZN Linux 2? 

          Mark Waite added a comment -

          I believe that Amazon Linux 2 is based on systemd, so it can start with either a service command or with systemctl

          Mark Waite added a comment - I believe that Amazon Linux 2 is based on systemd, so it can start with either a service command or with systemctl

            Unassigned Unassigned
            abhi2v Abhinav Varshney
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: