Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67635

Using "||" in pipeline targeting cloud labels cause multiple agents to be spun incessantly

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • core
    • Jenkins 2.319.2 on docker (jenkins/jenkins:2.319.2-jdk8) on Ubuntu 20.04

      Any of the following plugins::
      EC2 Cloud 1.66
      Anka-Build 2.7.0
    • 2.333,2.332.1

    Description

      General:

      Using "label1 || label2" in a pipeline, with the labels targeting cloud labels, is kicking off multiple agents of that cloud until one is able to have an executor.

      Testing process:

      I have tested this with two different cloud plugins (EC2 and Anka Build) and repeated the same steps for version 2.319.1 to make sure this is a Core issue and not a plugin issue.

      Step by step reproducing (with EC2 plugin):

      1. Install fresh Jenkins 2.319.2
      2. Install EC2 Plugin 1.66
      3. Set up two different labels in the EC2 Cloud
      4. Validate both of them can run properly (security groups, roles, keys, whatever is needed for your AWS)
      5. Create a basic pipeline job with this code:
        1. pipeline { agent { node { label 'label1 || label2' } }
           stages { stage('sleeping') { steps {
               sleep 9999999
           } } } }
          
        2. replace "label1" and "label2" with the two labels you set up in the cloud

      You can replace the sleep with whatever you want.

      You can do the same with the anka-build plugin. 

      Behavior Observed:

      EC2 Plugin: spinning up instances of only label1 non-stop. I aborted the job after 6 since these cost money. Screenshot attached.

      Anka plugin: Multiplle VMs are spinning up until the first one connects.

      Behavior +Observed on 2.319.1:+

      EC2 Plugin: a single instance from each label is spun up.

      Anka: A single VM is spun up.

      Attachments:

      ++EC2 console before running the job, after running the job on 2.319.2 and after running it again on 2.319.1.

       

      Attachments

        1. 1_before.png
          1_before.png
          38 kB
        2. 2_2.319.2.png
          2_2.319.2.png
          125 kB
        3. 3_2.319.1.png
          3_2.319.1.png
          135 kB

        Issue Links

          Activity

            danielbeck Daniel Beck added a comment - niv_keidan_veertu Thanks!
            danielbeck Daniel Beck added a comment - Initial report at https://groups.google.com/g/jenkinsci-dev/c/lyFon8KmP8I/m/gfcWXgmnAQAJ
            jglick Jesse Glick added a comment -

            Likely a regression from JENKINS-67099; CC vlatombe.

            jglick Jesse Glick added a comment - Likely a regression from JENKINS-67099 ; CC vlatombe .
            harryr7 Harry King added a comment - Also possibly related https://issues.jenkins.io/browse/JENKINS-67558  

            We are observing a similar issue since the update to 2.319.2 that sounds very related. Maybe someone with more insight can confirm it.

            We are using the ECS plug-in for starting Docker containers in AWS ECS. Since a few days (roughly correlated with the update from 2.319.1 to 2.319.2) we observe that an ECS agent is started and sitting idle but the job is not allocated to to it. Instead the ECS plug-in spins up another container and another container until eventually after several minutes (sometimes) the job is allocated on of the idle ECS nodes. We are not using || in our node labels but only && but the overall (mis)behaviour sounds very similar.

            sithmein Thorsten Meinl added a comment - We are observing a similar issue since the update to 2.319.2 that sounds very related. Maybe someone with more insight can confirm it. We are using the ECS plug-in for starting Docker containers in AWS ECS. Since a few days (roughly correlated with the update from 2.319.1 to 2.319.2) we observe that an ECS agent is started and sitting idle but the job is not allocated to to it. Instead the ECS plug-in spins up another container and another container until eventually after several minutes (sometimes) the job is allocated on of the idle ECS nodes. We are not using || in our node labels but only && but the overall (mis)behaviour sounds very similar.

            People

              niv_keidan_veertu niv keidan
              niv_keidan_veertu niv keidan
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: