Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32986

hard killing a pipeline leaves the JVM CPS thread running.

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • workflow-cps-plugin
    • None
    • pipeline 1.13
      jenkins 1.642.1

      In the event a pipeline build will not die you can hard kill it - however hard killing it will leave the JVMs CPS thread still running on the master.

      e.g. with the script

      def spin() {
    while (true) {}
}

def map = [:]
map ["spin_it"] = { spin() } 
}
parallel map

      you will need to hard kill it to stop it (on windows at least) - but inspecting the JVM threads you can see the CPS thread is still running in a tight loop.
      A hard kill should probably (if it is safe without causing deadlocks elsewhere) brutally kill the thread as well. After a while you may run out of handles or other native resources due to the thread usage, meaning you need to restart Jenkins to get it working again.

          [JENKINS-32986] hard killing a pipeline leaves the JVM CPS thread running.

          James Nord created issue -
          James Nord made changes -
          Link New: This issue is blocking JENKINS-25550 [ JENKINS-25550 ]

          James Nord added a comment -

          Not sure it can be blocking something that is fixed but here goes.

          James Nord added a comment - Not sure it can be blocking something that is fixed but here goes.
          James Nord made changes -
          Description Original: In the event a workflow won;t die you can hard kill it - however hard killing it will leave the JVMs CPS thread still running on the master.

          e.g. with the workflow
          {{noformat}}
          def spin() {
              while (true) {}
          }

          def map = [:]
          map ["spin_it"] = { spin() }
          }
          parallel map
          {{noformat}}

          you will need to hard kill it to stop it (on windows at least) - but inspecting the JVM threads you can see the CPS thread is still running in a tight loop.
          A hard kill should probably (if it is safe without causing deadlocks elsewhere) brutally kill the thread as well. After a while you may run out of handles or other native resources due to the thread usage, meaning you need to restart Jenkins to get it working again.           		New: In the event a workflow won;t die you can hard kill it - however hard killing it will leave the JVMs CPS thread still running on the master.

          e.g. with the workflow
          {noformat}
          def spin() {
              while (true) {}
          }

          def map = [:]
          map ["spin_it"] = { spin() }
          }
          parallel map
          {noformat}

          you will need to hard kill it to stop it (on windows at least) - but inspecting the JVM threads you can see the CPS thread is still running in a tight loop.
          A hard kill should probably (if it is safe without causing deadlocks elsewhere) brutally kill the thread as well. After a while you may run out of handles or other native resources due to the thread usage, meaning you need to restart Jenkins to get it working again.
          Jesse Glick made changes -
          Description Original: In the event a workflow won;t die you can hard kill it - however hard killing it will leave the JVMs CPS thread still running on the master.

          e.g. with the workflow
          {noformat}
          def spin() {
              while (true) {}
          }

          def map = [:]
          map ["spin_it"] = { spin() }
          }
          parallel map
          {noformat}

          you will need to hard kill it to stop it (on windows at least) - but inspecting the JVM threads you can see the CPS thread is still running in a tight loop.
          A hard kill should probably (if it is safe without causing deadlocks elsewhere) brutally kill the thread as well. After a while you may run out of handles or other native resources due to the thread usage, meaning you need to restart Jenkins to get it working again.           		New: In the event a pipeline build will not die you can hard kill it - however hard killing it will leave the JVMs CPS thread still running on the master.

          e.g. with the script

          {code}
          def spin() {
              while (true) {}
          }

          def map = [:]
          map ["spin_it"] = { spin() }
          }
          parallel map
          {code}

          you will need to hard kill it to stop it (on windows at least) - but inspecting the JVM threads you can see the CPS thread is still running in a tight loop.
          A hard kill should probably (if it is safe without causing deadlocks elsewhere) brutally kill the thread as well. After a while you may run out of handles or other native resources due to the thread usage, meaning you need to restart Jenkins to get it working again.
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 168715 ] New: JNJira + In-Review [ 183239 ]
          Jesse Glick made changes -
          Link New: This issue is related to JENKINS-25623 [ JENKINS-25623 ]

          Jesse Glick added a comment -

          Picking up some stuff from JENKINS-25623:

          • If the CPS VM is running native code, Thread.interrupt should be called. It should be given a limited grace period—say, a few seconds—to terminate; after that, resort to Thread.stop, making sure we are able to provide a fresh Thread for the pool so we can still run finally blocks or whatever.
          • We may also need some sort of per-build CPS VM CPU quota, distinct from timeout in that we do not care about wall clock time spent running a shell script on an agent, we just care about not overloading the master. Alternately, if a given build starts taking too much CPU time (measurable via System.nanoTime around runNextChunk), gradually being delaying its chunk execution (i.e., CpsThreadGroup.scheduleRun may call schedule rather than submit) so that it does not hog the system, and also institute a hard time limit for individual chunks (such as slow native methods).

          Jesse Glick added a comment - Picking up some stuff from JENKINS-25623 : If the CPS VM is running native code, Thread.interrupt should be called. It should be given a limited grace period—say, a few seconds—to terminate; after that, resort to Thread.stop , making sure we are able to provide a fresh Thread for the pool so we can still run finally blocks or whatever. We may also need some sort of per-build CPS VM CPU quota, distinct from timeout in that we do not care about wall clock time spent running a shell script on an agent, we just care about not overloading the master. Alternately, if a given build starts taking too much CPU time (measurable via System.nanoTime around runNextChunk ), gradually being delaying its chunk execution (i.e., CpsThreadGroup.scheduleRun may call schedule rather than submit ) so that it does not hog the system, and also institute a hard time limit for individual chunks (such as slow native methods).
          Jesse Glick made changes -
          Link New: This issue is related to JENKINS-37719 [ JENKINS-37719 ]
          Andrew Bayer made changes -
          Component/s New: pipeline-general [ 21692 ]

            jglick Jesse Glick
            teilo James Nord
            Votes:
            4 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: