Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67487

Start Jenkins command starts but does not return back to shell on Amazon Linux 2

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Blocker
    • Resolution: Fixed
    • packaging
    • None
    • Jenkins server 2.319.1
    • 2.333

    Description

      After upgrade to 2.319.1 (from 2.277.2), the command "service jenkins start" on the Linux console hangs (we waited for ~30 mins). It doesnt return back to shell. This currently prevents the ansible playbook from progressing since it gets stuck at this task waiting for a return to shell.

      previously, the 'service jenkins start' command should used to return to shell with 5 seconds.

      Eg:

      service jenkins start

      [ABC@ip-ABC]# service jenkins start
      Starting Jenkins
       
      Session terminated, killing shell... ...killed.
      [ABC@ ~]#
      

      Debugging the init.d script for jenkins, the daemon command which used to return back to the shell, doesnt do that anymore.

      echo -n "Starting Jenkins "
              daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" "$JAVA_CMD" $PARAMS > /dev/null
              RETVAL=$?
      

      Values of the variables:

      JENKINS_USER = jenkins

      JENKINS_PID_FILE = /var/run/jenkins.pid

      JAVA_CMD = /etc/alternatives/java

      PARAMS =  -Xmx3883m -Xms3883m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dhudson.model.DirectoryBrowserSupport.CSP="default-src 'self'; connect-src 'self' 'unsafe-inline' storybook.js.org; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self' 'unsafe-inline';" -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war and --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=8080 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

       

      We see that while the service start command is stuck, it is able to start Jenkins successfully. 

      We confirmed:

      1. Jenkins process (ps output) is up.
      2. /var/log/jenkins/jenkins.log has "Jenkins is fully up and running"

      Attachments

        Activity

          666999 nick a added a comment -

          markewaite

          i ve changed it no 0, but nothing happened, ive killed the process then started it via systemctl and it stuck again but jenkins is up and running so i made another change commented the RETVAL=0 total is the same 

          RETVAL=0

          case "$1" in
          start)
          echo -n "Starting Jenkins "
          daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" $JAVA_CMD $PARAMS > /dev/null
          RETVAL=0
          echo -n "RETVAL"
          if [ $RETVAL = 0 ]; then
          success
          echo > "$JENKINS_PID_FILE" # just in case we fail to find it
          MY_SESSION_ID=`/bin/ps h -o sess -p $$`
          echo -n "my sesion ID"

          1. get PID
            /bin/ps hww -u "$JENKINS_USER" -o sess,ppid,pid,cmd | \
            while read sess ppid pid cmd; do
            echo -n "while start"
            [ "$ppid" = 1 ] || continue
          2. this test doesn't work because Jenkins sets a new Session ID
          3. [ "$sess" = "$MY_SESSION_ID" ] || continue
            echo "$cmd" | grep $JENKINS_WAR > /dev/null
            [ $? = 0 ] || continue
            echo -n "before found PID"
          4. found a PID
            echo $pid > "$JENKINS_PID_FILE"
            echo -n $pid
            done
          666999 nick a added a comment - markewaite i ve changed it no 0, but nothing happened, ive killed the process then started it via systemctl and it stuck again but jenkins is up and running so i made another change commented the RETVAL=0 total is the same  RETVAL=0 case "$1" in start) echo -n "Starting Jenkins " daemon --user "$JENKINS_USER" --pidfile "$JENKINS_PID_FILE" $JAVA_CMD $PARAMS > /dev/null RETVAL=0 echo -n "RETVAL" if [ $RETVAL = 0 ]; then success echo > "$JENKINS_PID_FILE" # just in case we fail to find it MY_SESSION_ID=`/bin/ps h -o sess -p $$` echo -n "my sesion ID" get PID /bin/ps hww -u "$JENKINS_USER" -o sess,ppid,pid,cmd | \ while read sess ppid pid cmd; do echo -n "while start" [ "$ppid" = 1 ] || continue this test doesn't work because Jenkins sets a new Session ID [ "$sess" = "$MY_SESSION_ID" ] || continue echo "$cmd" | grep $JENKINS_WAR > /dev/null [ $? = 0 ] || continue echo -n "before found PID" found a PID echo $pid > "$JENKINS_PID_FILE" echo -n $pid done
          666999 nick a added a comment -

          markewaite

          this is the error itself 

          666999 nick a added a comment - markewaite this is the error itself 
          markewaite Mark Waite added a comment -

          That error seems to indicate that you have other customization included in your Jenkins service unit. Fixing those messages seems like a good place to start.

          markewaite Mark Waite added a comment - That error seems to indicate that you have other customization included in your Jenkins service unit. Fixing those messages seems like a good place to start.
          666999 nick a added a comment -

          markewaite 

          actually i got those configurations from ex devops, and have not made any changes, so here are they,

          do you think i should make any changes in it ?

          666999 nick a added a comment - markewaite   actually i got those configurations from ex devops, and have not made any changes, so here are they, do you think i should make any changes in it ?
          markewaite Mark Waite added a comment -

          666999 I'm not a systemd expert. You could compare those settings to a fresh installation to decide if you need to consider changes.

          markewaite Mark Waite added a comment - 666999 I'm not a systemd expert. You could compare those settings to a fresh installation to decide if you need to consider changes.

          People

            Unassigned Unassigned
            abhi2v Abhinav Varshney
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: