Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67558

Jenkins stuck "Waiting for next available executor" for agents that have been launched after a job starts

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • ec2-plugin
    • jenkins/jenkins:2.325-jdk8 docker image
      Amazon EC2 plugin Version 1.66
      Amazon Web Services SDK :: EC2 Version 1.12.101-300.vc09c7be9cb57

      I use the EC2 plugin (with no delay provisioning) to spin up nodes for my pipeline jobs.  I've noticed in either a recent plugin or Jenkins update (last 3 months), that when a new node is needed, there often seems to be a delay after the node is launched before the job will notice it can continue.  

      For example:

      • Master idle with no nodes running
      • Git commit triggers build
      • Job triggers creation of an EC2 instance (30 seconds).
      • Node init runs (2 minutes).
      • Job should now continue - agent now shows a build queue, but can sit idle for several minutes.  Checking the log output of the job shows an old message of "Waiting for next available executor" with a timestamp relating to before the node was requested.

      If another job needing the same label is triggered in the mean time, this appears to remind the other job that it was waiting for a node.

      Not terrible, but can double some build times, other jobs spin up more than one node during the build process.  Once the node is running, jobs seem to trigger as expected.

          [JENKINS-67558] Jenkins stuck "Waiting for next available executor" for agents that have been launched after a job starts

          Harry King added a comment -

          A small spot of further troubleshooting, in the Jenkins log, I can see:

          INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 1 computers, waiting for 1 more
          INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more
          INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more

          repeated every 10 seconds.  It's like the poll occurs every 10 seconds and we can see the new computer is spotted, but the condition it's waiting for is broken for freshly spun nodes.

          Harry King added a comment - A small spot of further troubleshooting, in the Jenkins log, I can see: INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 1 computers, waiting for 1 more INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more repeated every 10 seconds.  It's like the poll occurs every 10 seconds and we can see the new computer is spotted, but the condition it's waiting for is broken for freshly spun nodes.

          Justin added a comment - - edited

          I was directed here by OP commenting on my r/jenkinsci post where I'm experiencing similar behavior with two items in the build queue in spite of four available nodes with the desired label.  For anyone interested, my case is detailed here.  OP and I called out the same two plugins, and the behavior sounds the same to me.  I will be interested to hear of the outcome.

          Justin added a comment - - edited I was directed here by OP commenting on my r/jenkinsci post where I'm experiencing similar behavior with two items in the build queue in spite of four available nodes with the desired label.  For anyone interested, my case is detailed here .  OP and I called out the same two plugins, and the behavior sounds the same to me.  I will be interested to hear of the outcome.

          Harry King added a comment -

          Sounds like it could be related to the behaviour discussed in this change, which altered the behaviour in 2.319.2

          https://github.com/jenkinsci/jenkins/pull/6193

          Harry King added a comment - Sounds like it could be related to the behaviour discussed in this change, which altered the behaviour in 2.319.2 https://github.com/jenkinsci/jenkins/pull/6193

          Tim Jacomb added a comment - - edited

          I believe this is a duplicate of JENKINS-67635.

          If it's still an issue after the fix is released then please re-open / could you try https://github.com/jenkinsci/jenkins/pull/6193 and see if it fixes it for you

          Tim Jacomb added a comment - - edited I believe this is a duplicate of JENKINS-67635 . If it's still an issue after the fix is released then please re-open / could you try https://github.com/jenkinsci/jenkins/pull/6193 and see if it fixes it for you

            thoulen FABRIZIO MANFREDI
            harryr7 Harry King
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: