Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9185

ProcessTreeKiller / BUILD_ID from Maven task

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Incomplete
    • core
    • None
    • Linux Fedora/CentOS/RHEL

    Description

      Configuration is :

      Master Jenkins on Linux (launched with -Dhudson.util.ProcessTreeKiller.disable=true)
      Slaves Jenkins agents on Linux, started by java -jar slave.jar

      On Slave agents we have Maven 2/3 jobs where bash script is embedded to start/stop/patch Tomcat applications.
      To avoid detached (forked) tomcat to be destroyed, we used the BUILD_ID trick on the bash script.

      After upgrading from 1.397 to 1.403, we notice that jobs on slaves where killed.
      Same jobs started on Master Jenkins didn't have this auto-kill problem.
      Free style jobs on Master or Slave didn't see the problem, so the BUILD_ID still worked.

      To restore the correct behaviour, we add -Dhudson.util.ProcessTreeKiller.disable=true on start command line ie :

      java -jar slave.jar -Dhudson.util.ProcessTreeKiller.disable=true
      

      I suspect a problem in env vars propagation between Maven 2 or 3 and Jenkins somewhere between 1.397 and 1.403 since now the BUILD_ID env var set by the bash script called by Maven isn't seen by Hudson.

      Attachments

        Activity

          hgomez Henri Gomez created issue -

          Can you verify that your Tomcat process really isn't getting the BUILD_ID environment variable? Or can you provide a test case?

          I read the relevant part of the code, but the way it's designed makes it somewhat unlikely that this is caused by Maven/Freestyle difference.

          kohsuke Kohsuke Kawaguchi added a comment - Can you verify that your Tomcat process really isn't getting the BUILD_ID environment variable? Or can you provide a test case? I read the relevant part of the code, but the way it's designed makes it somewhat unlikely that this is caused by Maven/Freestyle difference.
          hgomez Henri Gomez added a comment -

          Bash script starting the Tomcat take care of BUILD_ID :

                      # Hack to avoid Hudson to kill spawned job (
                      # http://hudson.361315.n4.nabble.com/Hudson-killing-background-server-process-even-with-daemonize-td384633.html
                      if [ ! -z "$WORKSPACE" ]; then
                          export BUILD_ID=dontKillMe
                      fi
          

          I think WORKSPACE env var is set in both local and slave agents isn't it ?

          hgomez Henri Gomez added a comment - Bash script starting the Tomcat take care of BUILD_ID : # Hack to avoid Hudson to kill spawned job ( # http: //hudson.361315.n4.nabble.com/Hudson-killing-background-server-process-even-with-daemonize-td384633.html if [ ! -z "$WORKSPACE" ]; then export BUILD_ID=dontKillMe fi I think WORKSPACE env var is set in both local and slave agents isn't it ?

          I experience the same problem on Ubuntu with Jenkins 1.409 - JBoss process that I start from a Maven2 build is killed when the build is finished. I upgraded from Hudson 1.377 where it worked and spawned processes were not killed. I still use BUILD_ID=dontKillMe that I set on the spawned JBoss start script. During the time JBoss is running I verified that it really has that env variable, while the parent maven project has BUILD_ID value generated by Jenkins.

          zdenek Zdeněk Zikán added a comment - I experience the same problem on Ubuntu with Jenkins 1.409 - JBoss process that I start from a Maven2 build is killed when the build is finished. I upgraded from Hudson 1.377 where it worked and spawned processes were not killed. I still use BUILD_ID=dontKillMe that I set on the spawned JBoss start script. During the time JBoss is running I verified that it really has that env variable, while the parent maven project has BUILD_ID value generated by Jenkins.
          hgomez Henri Gomez added a comment - - edited

          Hi guys.

          I double check my scrip and I've got (without any tests now )

          export BUILD_ID=dontKillMe

          In our new settings, we're using slaves controlled by master via ssh, so I set the -Dhudson.util.ProcessTreeKiller.disable=true via JVM opts in slave definition

          hgomez Henri Gomez added a comment - - edited Hi guys. I double check my scrip and I've got (without any tests now ) export BUILD_ID=dontKillMe In our new settings, we're using slaves controlled by master via ssh, so I set the -Dhudson.util.ProcessTreeKiller.disable=true via JVM opts in slave definition
          hgomez Henri Gomez added a comment -

          Any news about this one ?
          How could I help to fix it ?

          hgomez Henri Gomez added a comment - Any news about this one ? How could I help to fix it ?
          hbjastad hbjastad added a comment -

          We've been running 1.377 for a loooong time now, because of this problem. Today, we upgraded to 1.413, and the problem still exists

          Is there anything we can do to provide more input towards getting it fixed?
          Or are we doomed to downgrade again and stay on 1.377 forever...?

          hbjastad hbjastad added a comment - We've been running 1.377 for a loooong time now, because of this problem. Today, we upgraded to 1.413, and the problem still exists Is there anything we can do to provide more input towards getting it fixed? Or are we doomed to downgrade again and stay on 1.377 forever...?
          hbjastad hbjastad added a comment -

          We start JBoss using maven-ant-plugin, with the following configuration:

          <configuration>
          	<tasks>
          		<exec executable="${jboss6x.home}/bin/run.sh" spawn="true">
          			<env key="BUILD_ID" value="dontKillMe" />
          			<arg value="-c" />
          			<arg value="${mds.server.name}" />
          		</exec>
          	</tasks>
          </configuration>
          

          And I modified the JBoss script run.sh, to print out the value of BUILD_ID:

          [exec]   BUILD_ID: dontKillMe
          

          (Note: I had to remove the spawn="true", to get the JBoss output)

          hbjastad hbjastad added a comment - We start JBoss using maven-ant-plugin, with the following configuration: <configuration> <tasks> <exec executable="${jboss6x.home}/bin/run.sh" spawn="true"> <env key="BUILD_ID" value="dontKillMe" /> <arg value="-c" /> <arg value="${mds.server.name}" /> </exec> </tasks> </configuration> And I modified the JBoss script run.sh, to print out the value of BUILD_ID: [exec] BUILD_ID: dontKillMe (Note: I had to remove the spawn="true", to get the JBoss output)
          hbjastad hbjastad added a comment -

          OK, so I tried putting it into JBoss' run.sh script, just to make sure it's not a problem between Maven/Ant:

          export BUILD_ID=dontKillMe
          

          The result is the same...
          So, there's no doubt that this is a bug in Hudson/Jenkins - introduced after 1.377 and still present in 1.413.

          hbjastad hbjastad added a comment - OK, so I tried putting it into JBoss' run.sh script, just to make sure it's not a problem between Maven/Ant: export BUILD_ID=dontKillMe The result is the same... So, there's no doubt that this is a bug in Hudson/Jenkins - introduced after 1.377 and still present in 1.413.
          hbjastad hbjastad added a comment -

          Last attempt to get 1.413 to leave our JBoss process alone, we set -Dhudson.util.ProcessTree.disable=true on the slave, as documented here: https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller

          And that works

          At least we have a workaround. Unfortunately, it means we have to disable process tree killer for all jobs on a given slave - instead of controlling it only for individual jobs, or even subprocesses of each job...

          hbjastad hbjastad added a comment - Last attempt to get 1.413 to leave our JBoss process alone, we set -Dhudson.util.ProcessTree.disable=true on the slave, as documented here: https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller And that works At least we have a workaround. Unfortunately, it means we have to disable process tree killer for all jobs on a given slave - instead of controlling it only for individual jobs, or even subprocesses of each job...
          hgomez Henri Gomez added a comment -

          I confirm we should also define -Dhudson.util.ProcessTree.disable=true on master and slaves.
          It's very problematic for us since for many jobs we need Jenkins to kill lone processes.
          A serious show stopper for us.

          hgomez Henri Gomez added a comment - I confirm we should also define -Dhudson.util.ProcessTree.disable=true on master and slaves. It's very problematic for us since for many jobs we need Jenkins to kill lone processes. A serious show stopper for us.
          hbjastad hbjastad added a comment -

          ping
          Is there any chance of finding a solution to this? It's a pity being stuck on 1.377

          hbjastad hbjastad added a comment - ping Is there any chance of finding a solution to this? It's a pity being stuck on 1.377
          danielbeck Daniel Beck added a comment -

          Does this issue still occur on recent Jenkins versions? Is there a simply script that when run in a build shows the problem so anyone investigating this doesn't have to deploy Tomcat from Maven?

          danielbeck Daniel Beck added a comment - Does this issue still occur on recent Jenkins versions? Is there a simply script that when run in a build shows the problem so anyone investigating this doesn't have to deploy Tomcat from Maven?
          danielbeck Daniel Beck added a comment -

          Resolving as 'Incomplete' after no response to comment asking for updated information in over two weeks.

          Due to the age of this issue, please file a new issue if this still occurs on recent Jenkins versions.

          danielbeck Daniel Beck added a comment - Resolving as 'Incomplete' after no response to comment asking for updated information in over two weeks. Due to the age of this issue, please file a new issue if this still occurs on recent Jenkins versions.
          danielbeck Daniel Beck made changes -
          Field Original Value New Value
          Resolution Incomplete [ 4 ]
          Status Open [ 1 ] Resolved [ 5 ]

          For further reference: I had exactly the same issue when running jobs on a Windows XP x64 slave. I had to unset HUDSON_COOKIE, HUDSON_SERVER_COOKIE, JENKINS_COOKIE or JENKINS_SERVER_COOKIE (I unset all of them, didn't check which one actually does the trick – JENKINS_COOKIE did not appear to be set though). We are running version 1.559, and the slave starts through JNLP (because we need it to access the GUI).

          I found the solution based on this StackOverflow question referencing a comment on the Spawning process from build wiki page.

          Notice that Jenkins continues to say that the process leaked file descriptors but at least it does not kill my background process any more (HSQLDB).

          didierl Didier Loiseau added a comment - For further reference: I had exactly the same issue when running jobs on a Windows XP x64 slave. I had to unset HUDSON_COOKIE, HUDSON_SERVER_COOKIE, JENKINS_COOKIE or JENKINS_SERVER_COOKIE (I unset all of them, didn't check which one actually does the trick – JENKINS_COOKIE did not appear to be set though). We are running version 1.559, and the slave starts through JNLP (because we need it to access the GUI). I found the solution based on this StackOverflow question referencing a comment on the Spawning process from build wiki page . Notice that Jenkins continues to say that the process leaked file descriptors but at least it does not kill my background process any more (HSQLDB).
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 139341 ] JNJira + In-Review [ 188451 ]

          People

            Unassigned Unassigned
            hgomez Henri Gomez
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: