Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53158

Pipelines leak OneOffExecutors when triggered on a temporary offline master

      When a Pipeline is triggered while the master node has been put temporarily offline, a OneOff (flyweight) executor is leaked.

      As I'm unsure whether the issue lies in core or in the pipelines at put both components.

       

      Environment

      • Version: 2.121.3
      • workflow-aggregator:2.5
      • workflow-cps:2.54

       
      Steps to reproduce

      • Create a simple Pipeline Job, eg:
      node('master') {
          sh 'echo hello'
      }
      • Put the master node temporary offline
      • Trigger the Pipeline build
      • Put the master node back online
      • Observe the leaked executor

       

      Observations

      This executor has the following characteristics:

      Executor #-1(-1)
       Active? false
       Likely Stuck? false
       Interrupted? false
       Busy? false
       Owner? hudson.model.Hudson$MasterComputer@228777fd
       Current Work Unit? null
       Causes Of Interruption? []
       Idle Start Milliseconds? 1534847122799
       Progress: -1
       Asynchronous Execution: null
       Executable: null
       executorOwner: hudson.model.Hudson$MasterComputer@228777fd
        tiedJobs: []

      Jenkins logs show:

      Executor #-1 for master grabbed hudson.model.queue.WorkUnit@4891a81f[work=hello] from queue but master went off-line before the task's worker thread started. No termination trace available.

      Pipeline logs show nothing:

      Démarré par l'utilisateur admin
      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] node
      Running on Jenkins in /Users/pierrebeitz/cbsupporthome/jenkins-home/workspace/hello
      [Pipeline] {
      [Pipeline] sh
      [hello] Running shell script
      + echo hello
      hello
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

      Workaround

      A restart of Jenkins will cleanup the executors.

       

       

       

          [JENKINS-53158] Pipelines leak OneOffExecutors when triggered on a temporary offline master

          Pierre Beitz created issue -
          Pierre Beitz made changes -
          Description Original: When a Pipeline is triggered while the master node has been put temporarily offline, a OneOff (flyweight) executor is leaked.

          As I'm unsure whether the issue lies in core or in the pipelines at put both components.

           

          *Environment*
           * Version: 2.121.3
           * workflow-aggregator:2.5
           * workflow-cps:2.54 'Pipeline: Groovy'

           
          *Steps to reproduce*
           * Create a simple Pipeline Job, eg:

          {code:java}
          node('master') {
              sh 'echo hello'
          }{code}
           * Put the master node temporary offline
           * Trigger the Pipeline build
           * Put the master node back online
           * Observe the leaked executor

           

          *Observations*

          This executor has the following characteristics:
          {code:java}
          Executor #-1(-1)
           Active? false
           Likely Stuck? false
           Interrupted? false
           Busy? false
           Owner? hudson.model.Hudson$MasterComputer@228777fd
           Current Work Unit? null
           Causes Of Interruption? []
           Idle Start Milliseconds? 1534847122799
           Progress: -1
           Asynchronous Execution: null
           Executable: null
           executorOwner: hudson.model.Hudson$MasterComputer@228777fd
            tiedJobs: []{code}
          Jenkins logs show:
          {noformat}
          Executor #-1 for master grabbed hudson.model.queue.WorkUnit@4891a81f[work=hello] from queue but master went off-line before the task's worker thread started. No termination trace available.{noformat}
          Pipeline logs show nothing:
          {noformat}
          Démarré par l'utilisateur admin
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] node
          Running on Jenkins in /Users/pierrebeitz/cbsupporthome/jenkins-home/workspace/hello
          [Pipeline] {
          [Pipeline] sh
          [hello] Running shell script
          + echo hello
          hello
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS{noformat}
          *Workaround*

          A restart of Jenkins will cleanup the executors.

           

           

           
          New: When a Pipeline is triggered while the master node has been put temporarily offline, a OneOff (flyweight) executor is leaked.

          As I'm unsure whether the issue lies in core or in the pipelines at put both components.

           

          *Environment*
           * Version: 2.121.3
           * workflow-aggregator:2.5
           * workflow-cps:2.54

           
           *Steps to reproduce*
           * Create a simple Pipeline Job, eg:

          {code:java}
          node('master') {
              sh 'echo hello'
          }{code}
           * Put the master node temporary offline
           * Trigger the Pipeline build
           * Put the master node back online
           * Observe the leaked executor

           

          *Observations*

          This executor has the following characteristics:
          {code:java}
          Executor #-1(-1)
           Active? false
           Likely Stuck? false
           Interrupted? false
           Busy? false
           Owner? hudson.model.Hudson$MasterComputer@228777fd
           Current Work Unit? null
           Causes Of Interruption? []
           Idle Start Milliseconds? 1534847122799
           Progress: -1
           Asynchronous Execution: null
           Executable: null
           executorOwner: hudson.model.Hudson$MasterComputer@228777fd
            tiedJobs: []{code}
          Jenkins logs show:
          {noformat}
          Executor #-1 for master grabbed hudson.model.queue.WorkUnit@4891a81f[work=hello] from queue but master went off-line before the task's worker thread started. No termination trace available.{noformat}
          Pipeline logs show nothing:
          {noformat}
          Démarré par l'utilisateur admin
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] node
          Running on Jenkins in /Users/pierrebeitz/cbsupporthome/jenkins-home/workspace/hello
          [Pipeline] {
          [Pipeline] sh
          [hello] Running shell script
          + echo hello
          hello
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS{noformat}
          *Workaround*

          A restart of Jenkins will cleanup the executors.

           

           

           
          Pierre Beitz made changes -
          Environment Original: * Version: 2.121.3
          * workflow-aggregator:2.5
          * workflow-cps:2.54 'Pipeline: Groovy'
          New: * Version: 2.121.3
          * workflow-aggregator:2.5
          * workflow-cps:2.54
          Andrew Bayer made changes -
          Component/s New: workflow-durable-task-step-plugin [ 21715 ]
          Component/s New: workflow-job-plugin [ 21716 ]
          Component/s Original: pipeline [ 21692 ]
          Devin Nusbaum made changes -
          Link New: This issue relates to JENKINS-45571 [ JENKINS-45571 ]
          Ryan Campbell made changes -
          Link New: This issue relates to JENKINS-57304 [ JENKINS-57304 ]
          Pierre Beitz made changes -
          Link New: This issue duplicates JENKINS-60348 [ JENKINS-60348 ]
          Pierre Beitz made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

            Unassigned Unassigned
            pierrebtz Pierre Beitz
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: