Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67558

Jenkins stuck "Waiting for next available executor" for agents that have been launched after a job starts

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Duplicate
    • ec2-plugin
    • jenkins/jenkins:2.325-jdk8 docker image
      Amazon EC2 plugin Version 1.66
      Amazon Web Services SDK :: EC2 Version 1.12.101-300.vc09c7be9cb57

    Description

      I use the EC2 plugin (with no delay provisioning) to spin up nodes for my pipeline jobs.  I've noticed in either a recent plugin or Jenkins update (last 3 months), that when a new node is needed, there often seems to be a delay after the node is launched before the job will notice it can continue.  

      For example:

      • Master idle with no nodes running
      • Git commit triggers build
      • Job triggers creation of an EC2 instance (30 seconds).
      • Node init runs (2 minutes).
      • Job should now continue - agent now shows a build queue, but can sit idle for several minutes.  Checking the log output of the job shows an old message of "Waiting for next available executor" with a timestamp relating to before the node was requested.

      If another job needing the same label is triggered in the mean time, this appears to remind the other job that it was waiting for a node.

      Not terrible, but can double some build times, other jobs spin up more than one node during the build process.  Once the node is running, jobs seem to trigger as expected.

      Attachments

        Issue Links

          Activity

            harryr7 Harry King added a comment -

            A small spot of further troubleshooting, in the Jenkins log, I can see:

            INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 1 computers, waiting for 1 more
            INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more
            INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more

            repeated every 10 seconds.  It's like the poll occurs every 10 seconds and we can see the new computer is spotted, but the condition it's waiting for is broken for freshly spun nodes.

            harryr7 Harry King added a comment - A small spot of further troubleshooting, in the Jenkins log, I can see: INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 1 computers, waiting for 1 more INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more INFO hudson.plugins.ec2.EC2Cloud#provision: We have now 2 computers, waiting for 0 more repeated every 10 seconds.  It's like the poll occurs every 10 seconds and we can see the new computer is spotted, but the condition it's waiting for is broken for freshly spun nodes.
            herter4171 Justin added a comment - - edited

            I was directed here by OP commenting on my r/jenkinsci post where I'm experiencing similar behavior with two items in the build queue in spite of four available nodes with the desired label.  For anyone interested, my case is detailed here.  OP and I called out the same two plugins, and the behavior sounds the same to me.  I will be interested to hear of the outcome.

            herter4171 Justin added a comment - - edited I was directed here by OP commenting on my r/jenkinsci post where I'm experiencing similar behavior with two items in the build queue in spite of four available nodes with the desired label.  For anyone interested, my case is detailed here .  OP and I called out the same two plugins, and the behavior sounds the same to me.  I will be interested to hear of the outcome.
            harryr7 Harry King added a comment -

            Sounds like it could be related to the behaviour discussed in this change, which altered the behaviour in 2.319.2

            https://github.com/jenkinsci/jenkins/pull/6193

            harryr7 Harry King added a comment - Sounds like it could be related to the behaviour discussed in this change, which altered the behaviour in 2.319.2 https://github.com/jenkinsci/jenkins/pull/6193
            timja Tim Jacomb added a comment - - edited

            I believe this is a duplicate of JENKINS-67635.

            If it's still an issue after the fix is released then please re-open / could you try https://github.com/jenkinsci/jenkins/pull/6193 and see if it fixes it for you

            timja Tim Jacomb added a comment - - edited I believe this is a duplicate of JENKINS-67635 . If it's still an issue after the fix is released then please re-open / could you try https://github.com/jenkinsci/jenkins/pull/6193 and see if it fixes it for you

            People

              thoulen FABRIZIO MANFREDI
              harryr7 Harry King
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: