Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47860

Jobs waiting on input, grab an executor after restart

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins 2.73.2
      pipeline-input-step-plugin 2.7

      We have several jobs that use the "input" pipeline step. This step is not inside a node block so they usually do not grab an executor when waiting for user input, as expected.

      We often have many of these waiting over our 200+ jobs which is not normally an issue.

      However, upon restarting (e.g. after upgrading plugins), these jobs resume and take up an executor. They appear in the left hand side and have logs like as follows:

      Deploy to prod?
      Proceed or Abort
      Resuming build at Tue Nov 07 12:12:56 GMT 2017 after Jenkins restart
      Waiting to resume part of Folder Name » Job Name » master #47: ???
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor
      Waiting to resume part of Folder Name » Job Name » master #47: Waiting for next available executor

      The current workaround is to manually abort each one but this time consuming and problematic if we do not want to abort them yet.

      Here is a possibly relevant portion of our shared library code, which gets called from most of our projects Jenkinsfiles:

      The relevant section of the code looks similar to this:

      for (def index = 0; index < allDeployEnvironments.size(); index++) {
        def nextEnvironment = ""
        if (index + 1 < allDeployEnvironments.size()) {
          nextEnvironment = allDeployEnvironments.get(index + 1)
        }
      
        //snip lines relating to deplpoyment.
      
        if (nextEnvironment) {   
          try {
            stage("${nextEnvironment} deployment prompt") {
              timeout(time: 30, unit: 'DAYS') {
              echo "Prompting for deploy to ${nextEnvironment}"
              input("Deploy to ${nextEnvironment}?")
            }
          }
        } catch (Exception ex) {
          echo "Input timed out"
          currentBuild.result = 'ABORTED'
          break
        }
      }

          [JENKINS-47860] Jobs waiting on input, grab an executor after restart

          Andrew Bayer added a comment -

          Can you include your Jenkinsfile please?

          Andrew Bayer added a comment - Can you include your Jenkinsfile please?

          Nicholas Boll added a comment -

          Same happens to us. input step not in a node block and works fine normally but on Jenkins restart takes a node executor

          Nicholas Boll added a comment - Same happens to us. input step not in a node block and works fine normally but on Jenkins restart takes a node executor

          Rupert Madden-Abbott added a comment - - edited

          Unfortunately, its been a bit difficult for me to recreate this issue.

          It's not a simple matter of providing a Jenkinsfile because the Jenkinsfiles reference a shared library which is where the input step is called.

          I've added a snippet to the ticket which should add more context, but strips back some of the hopefully irrelevant complexity.

          Rupert Madden-Abbott added a comment - - edited Unfortunately, its been a bit difficult for me to recreate this issue. It's not a simple matter of providing a Jenkinsfile because the Jenkinsfiles reference a shared library which is where the input step is called. I've added a snippet to the ticket which should add more context, but strips back some of the hopefully irrelevant complexity.

          Cédric Caron added a comment -

          We are facing the same issue.

          JENKINS-47860 and JENKINS-41791 currently make a CD pipeline based on Jenkins very brittle.

          Could those 2 bugs be moved to a higher level of criticity? 

          Cédric Caron added a comment - We are facing the same issue. JENKINS-47860  and JENKINS-41791 currently make a CD pipeline based on Jenkins very brittle. Could those 2 bugs be moved to a higher level of criticity? 

          cecaron_mn - are you able to provide a Jenkinsfile that replicates this issue? Part of the problem I've faced in trying to debug this issue is that I've not been able to replicate it reliably.

          Rupert Madden-Abbott added a comment - cecaron_mn - are you able to provide a Jenkinsfile that replicates this issue? Part of the problem I've faced in trying to debug this issue is that I've not been able to replicate it reliably.

          Cédric Caron added a comment -

          rupert It s not that straightforward as we are using a library that manage part of the pipeline but I will try to provide you something to reproduce.

          Cédric Caron added a comment - rupert  It s not that straightforward as we are using a library that manage part of the pipeline but I will try to provide you something to reproduce.

          cecaron_mn - I completely understand - its exactly the same situation for us as well. It would be interesting to see what similarities there are with the snippet I posted in the original bug report above.

          Rupert Madden-Abbott added a comment - cecaron_mn - I completely understand - its exactly the same situation for us as well. It would be interesting to see what similarities there are with the snippet I posted in the original bug report above.

          Arvid Arntzen added a comment -

          We experience the same issue with similar pipeline script containing input with timeout.

          It typically happens after reboot of master and the node that ran the last part of the pending builds waiting for input.

          All the builds waiting for input (that was not taking any executor before reboot) is now taking up all the executors on the node and are stuck there.

          Arvid Arntzen added a comment - We experience the same issue with similar pipeline script containing input with timeout. It typically happens after reboot of master and the node that ran the last part of the pending builds waiting for input. All the builds waiting for input (that was not taking any executor before reboot) is now taking up all the executors on the node and are stuck there.

            Unassigned Unassigned
            rupert Rupert Madden-Abbott
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: