Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53001

When using the same label 2+ times in a parallel block, plugin only launches one agent

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: amazon-ecs-plugin
    • Labels:
      None
    • Environment:
      amazon-ecs plugin version 1.16
    • Similar Issues:
    • Released As:
      v1.18

      Description

      If you have one agent template with labels defined, and you create a Pipeline job that requests nodes of the same label in a parallel block, only one ECS agent will be launched. As a result, the work is only done serially, rather than in parallel as expected, because only one executor is available. Example Pipeline code:

      def tasks = [:]
      
      tasks["task_1"] = {
        stage ("task_1"){    
          node('mylabel') {  
              sh 'echo foo'
          }
        }
      }
      tasks["task_2"] = {
        stage ("task_2"){    
          node('mylabel') {  
              sh 'echo bar'
          }
        }
      }
      
      parallel tasks
      

      If you run two copies of this job, or any other job that uses the same label, an agent will be launched for each job that is running. And if the parallel block of your Pipeline job refers to two different labels, then two agents will be launched. The issue is only with uses of the same label within the parallel branches.

        Attachments

          Activity

          Hide
          ddumont Dan Dumont added a comment - - edited

          "And if the parallel block of your Pipeline job refers to two different labels"

          Even if the different labels point to the same actual agent.

          Show
          ddumont Dan Dumont added a comment - - edited "And if the parallel block of your Pipeline job refers to two  different  labels" Even if the different labels point to the same actual agent.
          Hide
          pgarbe Philipp Garbe added a comment -

          The logic how agents are launched has changed in v1.18. It's still not completely in parallel, as there's a tradeoff between running completely in parallel and having started too many agents. 

          Jenkins calls `provision`, but even for parallel stages, it sets `excessWorkload` to 1. If the ECS task takes too long to start, `provision` is called again. This would lead to the situation that too many unnecessary agents are launched. That's why this code prevents further provisioning as long as another agent with the same label is in pending state.

          https://github.com/jenkinsci/amazon-ecs-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSCloud.java#L183-L198

          Show
          pgarbe Philipp Garbe added a comment - The logic how agents are launched has changed in v1.18. It's still not completely in parallel, as there's a tradeoff between running completely in parallel and having started too many agents.  Jenkins calls `provision`, but even for parallel stages, it sets `excessWorkload` to 1. If the ECS task takes too long to start, `provision` is called again. This would lead to the situation that too many unnecessary agents are launched. That's why this code prevents further provisioning as long as another agent with the same label is in pending state. https://github.com/jenkinsci/amazon-ecs-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSCloud.java#L183-L198

            People

            Assignee:
            pgarbe Philipp Garbe
            Reporter:
            owenmehegan Owen Mehegan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: