Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53158

Pipelines leak OneOffExecutors when triggered on a temporary offline master

    XMLWordPrintable

Details

    Description

      When a Pipeline is triggered while the master node has been put temporarily offline, a OneOff (flyweight) executor is leaked.

      As I'm unsure whether the issue lies in core or in the pipelines at put both components.

       

      Environment

      • Version: 2.121.3
      • workflow-aggregator:2.5
      • workflow-cps:2.54

       
      Steps to reproduce

      • Create a simple Pipeline Job, eg:
      node('master') {
          sh 'echo hello'
      }
      • Put the master node temporary offline
      • Trigger the Pipeline build
      • Put the master node back online
      • Observe the leaked executor

       

      Observations

      This executor has the following characteristics:

      Executor #-1(-1)
       Active? false
       Likely Stuck? false
       Interrupted? false
       Busy? false
       Owner? hudson.model.Hudson$MasterComputer@228777fd
       Current Work Unit? null
       Causes Of Interruption? []
       Idle Start Milliseconds? 1534847122799
       Progress: -1
       Asynchronous Execution: null
       Executable: null
       executorOwner: hudson.model.Hudson$MasterComputer@228777fd
        tiedJobs: []

      Jenkins logs show:

      Executor #-1 for master grabbed hudson.model.queue.WorkUnit@4891a81f[work=hello] from queue but master went off-line before the task's worker thread started. No termination trace available.

      Pipeline logs show nothing:

      Démarré par l'utilisateur admin
      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] node
      Running on Jenkins in /Users/pierrebeitz/cbsupporthome/jenkins-home/workspace/hello
      [Pipeline] {
      [Pipeline] sh
      [hello] Running shell script
      + echo hello
      hello
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS

      Workaround

      A restart of Jenkins will cleanup the executors.

       

       

       

      Attachments

        Issue Links

          Activity

            recampbell Ryan Campbell added a comment -

            Seems like behavior similar to JENKINS-57304

            recampbell Ryan Campbell added a comment - Seems like behavior similar to JENKINS-57304
            pierrebtz Pierre Beitz added a comment -

            Still reproducible:

            • Jenkins 2.204.1
            • workflow-aggregator:2.6
            • workflow-cps:2.78
            pierrebtz Pierre Beitz added a comment - Still reproducible: Jenkins 2.204.1 workflow-aggregator:2.6 workflow-cps:2.78
            pierrebtz Pierre Beitz added a comment -

            Further testing shows that https://issues.jenkins-ci.org/browse/JENKINS-60348 fixed it. Successfully tested in 2.211!

            pierrebtz Pierre Beitz added a comment - Further testing shows that  https://issues.jenkins-ci.org/browse/JENKINS-60348  fixed it. Successfully tested in 2.211!

            People

              Unassigned Unassigned
              pierrebtz Pierre Beitz
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: