Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67635

Using "||" in pipeline targeting cloud labels cause multiple agents to be spun incessantly

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Jenkins 2.319.2 on docker (jenkins/jenkins:2.319.2-jdk8) on Ubuntu 20.04

      Any of the following plugins::
      EC2 Cloud 1.66
      Anka-Build 2.7.0
    • 2.333,2.332.1

      General:

      Using "label1 || label2" in a pipeline, with the labels targeting cloud labels, is kicking off multiple agents of that cloud until one is able to have an executor.

      Testing process:

      I have tested this with two different cloud plugins (EC2 and Anka Build) and repeated the same steps for version 2.319.1 to make sure this is a Core issue and not a plugin issue.

      Step by step reproducing (with EC2 plugin):

      1. Install fresh Jenkins 2.319.2
      2. Install EC2 Plugin 1.66
      3. Set up two different labels in the EC2 Cloud
      4. Validate both of them can run properly (security groups, roles, keys, whatever is needed for your AWS)
      5. Create a basic pipeline job with this code:
        1. pipeline { agent { node { label 'label1 || label2' } }
           stages { stage('sleeping') { steps {
               sleep 9999999
           } } } }
          
        2. replace "label1" and "label2" with the two labels you set up in the cloud

      You can replace the sleep with whatever you want.

      You can do the same with the anka-build plugin. 

      Behavior Observed:

      EC2 Plugin: spinning up instances of only label1 non-stop. I aborted the job after 6 since these cost money. Screenshot attached.

      Anka plugin: Multiplle VMs are spinning up until the first one connects.

      Behavior +Observed on 2.319.1:+

      EC2 Plugin: a single instance from each label is spun up.

      Anka: A single VM is spun up.

      Attachments:

      ++EC2 console before running the job, after running the job on 2.319.2 and after running it again on 2.319.1.

       

        1. 1_before.png
          1_before.png
          38 kB
        2. 2_2.319.2.png
          2_2.319.2.png
          125 kB
        3. 3_2.319.1.png
          3_2.319.1.png
          135 kB

          [JENKINS-67635] Using "||" in pipeline targeting cloud labels cause multiple agents to be spun incessantly

          Tim Jacomb added a comment -

          Tim Jacomb added a comment - Possibly related https://github.com/jenkinsci/docker/issues/1274

          niv keidan added a comment -

          timja I agree, looks the same.

          PR opened for fix https://github.com/jenkinsci/jenkins/pull/6193

          niv keidan added a comment - timja  I agree, looks the same. PR opened for fix  https://github.com/jenkinsci/jenkins/pull/6193

          Daniel Beck added a comment -

          Create a basic pipeline job with this code

          The report is broken thanks to Jira. Could you put the Pipeline into a { noformat } block?

          Daniel Beck added a comment - Create a basic pipeline job with this code The report is broken thanks to Jira. Could you put the Pipeline into a { noformat } block?

          niv keidan added a comment -

          danielbeck that ok?

          niv keidan added a comment - danielbeck  that ok?

          Daniel Beck added a comment -

          Daniel Beck added a comment - niv_keidan_veertu Thanks!

          Daniel Beck added a comment -

          Daniel Beck added a comment - Initial report at https://groups.google.com/g/jenkinsci-dev/c/lyFon8KmP8I/m/gfcWXgmnAQAJ

          Jesse Glick added a comment -

          Likely a regression from JENKINS-67099; CC vlatombe.

          Jesse Glick added a comment - Likely a regression from JENKINS-67099 ; CC vlatombe .

          Harry King added a comment -

          Harry King added a comment - Also possibly related https://issues.jenkins.io/browse/JENKINS-67558  

          We are observing a similar issue since the update to 2.319.2 that sounds very related. Maybe someone with more insight can confirm it.

          We are using the ECS plug-in for starting Docker containers in AWS ECS. Since a few days (roughly correlated with the update from 2.319.1 to 2.319.2) we observe that an ECS agent is started and sitting idle but the job is not allocated to to it. Instead the ECS plug-in spins up another container and another container until eventually after several minutes (sometimes) the job is allocated on of the idle ECS nodes. We are not using || in our node labels but only && but the overall (mis)behaviour sounds very similar.

          Thorsten Meinl added a comment - We are observing a similar issue since the update to 2.319.2 that sounds very related. Maybe someone with more insight can confirm it. We are using the ECS plug-in for starting Docker containers in AWS ECS. Since a few days (roughly correlated with the update from 2.319.1 to 2.319.2) we observe that an ECS agent is started and sitting idle but the job is not allocated to to it. Instead the ECS plug-in spins up another container and another container until eventually after several minutes (sometimes) the job is allocated on of the idle ECS nodes. We are not using || in our node labels but only && but the overall (mis)behaviour sounds very similar.

            niv_keidan_veertu niv keidan
            niv_keidan_veertu niv keidan
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: