Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67487

Start Jenkins command starts but does not return back to shell on Amazon Linux 2

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • packaging
    • None
    • Jenkins server 2.319.1
    • 2.333

      After upgrade to 2.319.1 (from 2.277.2), the command "service jenkins start" on the Linux console hangs (we waited for ~30 mins). It doesnt return back to shell. This currently prevents the ansible playbook from progressing since it gets stuck at this task waiting for a return to shell.

      previously, the 'service jenkins start' command should used to return to shell with 5 seconds.

      Eg:

      service jenkins start

      [ABC@ip-ABC]# service jenkins start
      Starting Jenkins
       
      Session terminated, killing shell... ...killed.
      [ABC@ ~]#
      

      Debugging the init.d script for jenkins, the daemon command which used to return back to the shell, doesnt do that anymore.

      echo -n "Starting Jenkins "
              daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" "$JAVA_CMD" $PARAMS > /dev/null
              RETVAL=$?
      

      Values of the variables:

      JENKINS_USER = jenkins

      JENKINS_PID_FILE = /var/run/jenkins.pid

      JAVA_CMD = /etc/alternatives/java

      PARAMS =  -Xmx3883m -Xms3883m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dhudson.model.DirectoryBrowserSupport.CSP="default-src 'self'; connect-src 'self' 'unsafe-inline' storybook.js.org; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self' 'unsafe-inline';" -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war and --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

       

      We see that while the service start command is stuck, it is able to start Jenkins successfully. 

      We confirmed:

      1. Jenkins process (ps output) is up.
      2. /var/log/jenkins/jenkins.log has "Jenkins is fully up and running"

          [JENKINS-67487] Start Jenkins command starts but does not return back to shell on Amazon Linux 2

          Mark Waite added a comment -

          Thanks lg17031 for doing that research! Much appreciated.

          Would you like to submit a pull request to fix the issue?

          Mark Waite added a comment - Thanks lg17031 for doing that research! Much appreciated. Would you like to submit a pull request to fix the issue?

          Lee Greene added a comment -

          To further follow up on this Centos 7 this is from the journalctl output from the startup. Not sure what the return code is for the WARNING message, that could be what is not returning a 0 return code.

          Jan 11 14:24:43 instance jenkins[970]: + RETVAL=0
          Jan 11 14:24:43 instance jenkins[970]: + case "$1" in
          Jan 11 14:24:43 instance jenkins[970]: + echo -n 'Starting Jenkins '
          Jan 11 14:24:43 instance jenkins[970]: Starting Jenkins + daemon --user jenkins --pidfile /var/run/jenkins.pid /bin/java -Djava.awt.headless=true -Djenkins.install.runSetupWizard=false -Djava.net.preferIPv4Stack=true '-Dhudson.model.
          Jan 11 14:24:43 instance jenkins[970]: + local gotbase= force= nicelevel corelimit
          Jan 11 14:24:43 instance jenkins[970]: + local pid base= user= nice= bg= pid_file=
          Jan 11 14:24:43 instance jenkins[970]: + local cgroup=
          Jan 11 14:24:43 instance jenkins[970]: + nicelevel=0
          Jan 11 14:24:43 instance jenkins[970]: + '[' --user '!=' -user ']'
          Jan 11 14:24:43 instance jenkins[970]: + case $1 in
          Jan 11 14:24:43 instance jenkins[970]: + user=jenkins
          Jan 11 14:24:43 instance jenkins[970]: + shift 2
          Jan 11 14:24:43 instance jenkins[970]: + '[' --pidfile '!=' -pidfile ']'
          Jan 11 14:24:43 instance jenkins[970]: + case $1 in
          Jan 11 14:24:43 instance jenkins[970]: + pid_file=/var/run/jenkins.pid
          Jan 11 14:24:43 instance jenkins[970]: + shift 2
          Jan 11 14:24:43 instance jenkins[970]: + '[' /bin/java '!=' /bin/java ']'
          Jan 11 14:24:43 instance jenkins[970]: + '[' -z '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + base=java
          Jan 11 14:24:43 instance jenkins[970]: + __pids_var_run java /var/run/jenkins.pid
          Jan 11 14:24:43 instance jenkins[970]: + local base=java
          Jan 11 14:24:43 instance jenkins[970]: + local pid_file=/var/run/jenkins.pid
          Jan 11 14:24:43 instance jenkins[970]: ++ /usr/bin/dirname /var/run/jenkins.pid
          Jan 11 14:24:43 instance jenkins[970]: + local pid_dir=
          Jan 11 14:24:43 instance jenkins[970]: + local binary=
          Jan 11 14:24:43 instance jenkins[970]: + '[' -d '' -a '!' -r '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + pid=
          Jan 11 14:24:43 instance jenkins[970]: + '[' -f /var/run/jenkins.pid ']'
          Jan 11 14:24:43 instance jenkins[970]: + return 3
          Jan 11 14:24:43 instance jenkins[970]: + '[' -n '' -a -z '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + corelimit='ulimit -S -c 0'
          Jan 11 14:24:43 instance jenkins[970]: + '[' -n '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + '[' -n '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + '[' serial = verbose -a -z '' ']'
          Jan 11 14:24:43 instance jenkins[970]: + '[' -z jenkins ']'
          Jan 11 14:24:43 instance jenkins[970]: + runuser -s /bin/bash jenkins -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /bin/java -Djava.awt.headless=true -Djenkins.install.runSetupWizard=false -Djava.net.preferIPv4Stack=true -Dhudson.model.Direc
          Jan 11 14:24:49 instance jenkins[970]: WARNING: An illegal reflective access operation has occurred
          Jan 11 14:24:49 instance jenkins[970]: WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(ja
          Jan 11 14:24:49 instance jenkins[970]: WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
          Jan 11 14:24:49 instance jenkins[970]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
          Jan 11 14:24:49 instance jenkins[970]: WARNING: All illegal access operations will be denied in a future release
          Jan 11 14:29:43 instance systemd[1]: jenkins.service start operation timed out. Terminating.
          Jan 11 14:29:43 instance systemd[1]: Failed to start LSB: Jenkins Automation Server.

          Lee Greene added a comment - To further follow up on this Centos 7 this is from the journalctl output from the startup. Not sure what the return code is for the WARNING message, that could be what is not returning a 0 return code. Jan 11 14:24:43 instance jenkins [970] : + RETVAL=0 Jan 11 14:24:43 instance jenkins [970] : + case "$1" in Jan 11 14:24:43 instance jenkins [970] : + echo -n 'Starting Jenkins ' Jan 11 14:24:43 instance jenkins [970] : Starting Jenkins + daemon --user jenkins --pidfile /var/run/jenkins.pid /bin/java -Djava.awt.headless=true -Djenkins.install.runSetupWizard=false -Djava.net.preferIPv4Stack=true '-Dhudson.model. Jan 11 14:24:43 instance jenkins [970] : + local gotbase= force= nicelevel corelimit Jan 11 14:24:43 instance jenkins [970] : + local pid base= user= nice= bg= pid_file= Jan 11 14:24:43 instance jenkins [970] : + local cgroup= Jan 11 14:24:43 instance jenkins [970] : + nicelevel=0 Jan 11 14:24:43 instance jenkins [970] : + ' [' --user '!=' -user '] ' Jan 11 14:24:43 instance jenkins [970] : + case $1 in Jan 11 14:24:43 instance jenkins [970] : + user=jenkins Jan 11 14:24:43 instance jenkins [970] : + shift 2 Jan 11 14:24:43 instance jenkins [970] : + ' [' --pidfile '!=' -pidfile '] ' Jan 11 14:24:43 instance jenkins [970] : + case $1 in Jan 11 14:24:43 instance jenkins [970] : + pid_file=/var/run/jenkins.pid Jan 11 14:24:43 instance jenkins [970] : + shift 2 Jan 11 14:24:43 instance jenkins [970] : + ' [' /bin/java '!=' /bin/java '] ' Jan 11 14:24:43 instance jenkins [970] : + ' [' -z '' '] ' Jan 11 14:24:43 instance jenkins [970] : + base=java Jan 11 14:24:43 instance jenkins [970] : + __pids_var_run java /var/run/jenkins.pid Jan 11 14:24:43 instance jenkins [970] : + local base=java Jan 11 14:24:43 instance jenkins [970] : + local pid_file=/var/run/jenkins.pid Jan 11 14:24:43 instance jenkins [970] : ++ /usr/bin/dirname /var/run/jenkins.pid Jan 11 14:24:43 instance jenkins [970] : + local pid_dir= Jan 11 14:24:43 instance jenkins [970] : + local binary= Jan 11 14:24:43 instance jenkins [970] : + ' [' -d '' -a '!' -r '' '] ' Jan 11 14:24:43 instance jenkins [970] : + pid= Jan 11 14:24:43 instance jenkins [970] : + ' [' -f /var/run/jenkins.pid '] ' Jan 11 14:24:43 instance jenkins [970] : + return 3 Jan 11 14:24:43 instance jenkins [970] : + ' [' -n '' -a -z '' '] ' Jan 11 14:24:43 instance jenkins [970] : + corelimit='ulimit -S -c 0' Jan 11 14:24:43 instance jenkins [970] : + ' [' -n '' '] ' Jan 11 14:24:43 instance jenkins [970] : + ' [' -n '' '] ' Jan 11 14:24:43 instance jenkins [970] : + ' [' serial = verbose -a -z '' '] ' Jan 11 14:24:43 instance jenkins [970] : + ' [' -z jenkins '] ' Jan 11 14:24:43 instance jenkins [970] : + runuser -s /bin/bash jenkins -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /bin/java -Djava.awt.headless=true -Djenkins.install.runSetupWizard=false -Djava.net.preferIPv4Stack=true -Dhudson.model.Direc Jan 11 14:24:49 instance jenkins [970] : WARNING: An illegal reflective access operation has occurred Jan 11 14:24:49 instance jenkins [970] : WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 ( file:/var/cache/jenkins/war/WEB-INF/lib/guice-4.0.jar ) to method java.lang.ClassLoader.defineClass(ja Jan 11 14:24:49 instance jenkins [970] : WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2 Jan 11 14:24:49 instance jenkins [970] : WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Jan 11 14:24:49 instance jenkins [970] : WARNING: All illegal access operations will be denied in a future release Jan 11 14:29:43 instance systemd [1] : jenkins.service start operation timed out. Terminating. Jan 11 14:29:43 instance systemd [1] : Failed to start LSB: Jenkins Automation Server.

          Mark Waite added a comment -

          Jenkins 2.333 removes the dependency on the daemon program. It should resolve this issue.

          I checked that pull request 266 also resolves the issue as part of the switch from System V init to systemd.

          Mark Waite added a comment - Jenkins 2.333 removes the dependency on the daemon program. It should resolve this issue. I checked that pull request 266 also resolves the issue as part of the switch from System V init to systemd.

          nick a added a comment -

          lg17031 

          Hello facing the same issue on Centos 7, modified /etc/rc.d/init.d/jenkins,

          added some echo before RETVAL=$ but doesnt work for me, have you got any other ideas?

          nick a added a comment - lg17031   Hello facing the same issue on Centos 7, modified /etc/rc.d/init.d/jenkins, added some echo before RETVAL=$ but doesnt work for me, have you got any other ideas?

          Mark Waite added a comment -

          666999 as a short term workaround, you can use RETVAL=0 to assure that the script reports success.

          Mark Waite added a comment - 666999 as a short term workaround, you can use RETVAL=0 to assure that the script reports success.

          nick a added a comment -

          markewaite

          i ve changed it no 0, but nothing happened, ive killed the process then started it via systemctl and it stuck again but jenkins is up and running so i made another change commented the RETVAL=0 total is the same 

          RETVAL=0

          case "$1" in
          start)
          echo -n "Starting Jenkins "
          daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" $JAVA_CMD $PARAMS > /dev/null
          RETVAL=0
          echo -n "RETVAL"
          if [ $RETVAL = 0 ]; then
          success
          echo > "$JENKINS_PID_FILE" # just in case we fail to find it
          MY_SESSION_ID=`/bin/ps h -o sess -p $$`
          echo -n "my sesion ID"

          1. get PID
            /bin/ps hww -u "$JENKINS_USER" -o sess,ppid,pid,cmd | \
            while read sess ppid pid cmd; do
            echo -n "while start"
            [ "$ppid" = 1 ] || continue
          2. this test doesn't work because Jenkins sets a new Session ID
          3. [ "$sess" = "$MY_SESSION_ID" ] || continue
            echo "$cmd" | grep $JENKINS_WAR > /dev/null
            [ $? = 0 ] || continue
            echo -n "before found PID"
          4. found a PID
            echo $pid > "$JENKINS_PID_FILE"
            echo -n $pid
            done

          nick a added a comment - markewaite i ve changed it no 0, but nothing happened, ive killed the process then started it via systemctl and it stuck again but jenkins is up and running so i made another change commented the RETVAL=0 total is the same  RETVAL=0 case "$1" in start) echo -n "Starting Jenkins " daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" $JAVA_CMD $PARAMS > /dev/null RETVAL=0 echo -n "RETVAL" if [ $RETVAL = 0 ]; then success echo > "$JENKINS_PID_FILE" # just in case we fail to find it MY_SESSION_ID=`/bin/ps h -o sess -p $$` echo -n "my sesion ID" get PID /bin/ps hww -u "$JENKINS_USER" -o sess,ppid,pid,cmd | \ while read sess ppid pid cmd; do echo -n "while start" [ "$ppid" = 1 ] || continue this test doesn't work because Jenkins sets a new Session ID [ "$sess" = "$MY_SESSION_ID" ] || continue echo "$cmd" | grep $JENKINS_WAR > /dev/null [ $? = 0 ] || continue echo -n "before found PID" found a PID echo $pid > "$JENKINS_PID_FILE" echo -n $pid done

          nick a added a comment -

          markewaite

          this is the error itself 

          nick a added a comment - markewaite this is the error itself 

          Mark Waite added a comment -

          That error seems to indicate that you have other customization included in your Jenkins service unit. Fixing those messages seems like a good place to start.

          Mark Waite added a comment - That error seems to indicate that you have other customization included in your Jenkins service unit. Fixing those messages seems like a good place to start.

          nick a added a comment -

          markewaite 

          actually i got those configurations from ex devops, and have not made any changes, so here are they,

          do you think i should make any changes in it ?

          nick a added a comment - markewaite   actually i got those configurations from ex devops, and have not made any changes, so here are they, do you think i should make any changes in it ?

          Mark Waite added a comment -

          666999 I'm not a systemd expert. You could compare those settings to a fresh installation to decide if you need to consider changes.

          Mark Waite added a comment - 666999 I'm not a systemd expert. You could compare those settings to a fresh installation to decide if you need to consider changes.

            Unassigned Unassigned
            abhi2v Abhinav Varshney
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: