Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67990

Issues with declarative pipeline syntax, parallel and docker agents running on different labels

    XMLWordPrintable

Details

    Description

      When using pipeline combined with Docker, physical agents and the 'parallel' syntax the result produces unwanted behaviour.

      Some background for this piece of code:

      • linux_fleet are a group of instances a high limit of instances that are spun up when Jenkins requests their specific label
      • they all have 1 executor
      • they are able to scale up without issue in non-parallel builds

      Intended:
      Test1 and Test2 build at the same time on different physical agents within docker containers.

      Actual:
      It picks a random agent for any, then attempts to build & run docker containers on that top level agent. It does not spin up anything except the initial agent.

      pipeline {
          agent any
          stages {
              stage('Build Tests') {
                  parallel {
                      stage('Build Test1') {
                          agent {
                              docker {
                                  image 'debian:latest'
                                  label 'linux_fleet'
                                  alwaysPull true
                                  reuseNode true
                              }
                          }
                          steps {
                              sh """
                              touch test1.test
                          """
                          }
                      }
                      stage('Build Test2') {
                          agent {
                              docker {
                                  image 'debian:latest'
                                  label 'linux_fleet'
                                  alwaysPull true
                                  reuseNode true
                              }
                          }
                          steps {
                              sh """
                              touch test2.test
                          """
                          }
                      }
                  }
              }
          }
      }

      If I change this code to instead define only labels of physical agents and not use the Docker syntax and instead use 'withDockerContainer' to run the code, it results in more physical agents being spun up for each parallel build, but deadlocking issues actually starting the builds.

      agent any
      stage('Build Test') {
                  parallel {
                      stage('Build Test1') {
                          agent {label 'linux_fleet'}
                          steps {
                               withDockerContainer(url: "dockerurl") {
                                  sh """
                                      touch test1.test
                                  """
                              }
                          }
                      }
      

      A very related issue:

      https://issues.jenkins.io/browse/JENKINS-46831

       

       

      Attachments

        Activity

          People

            csanchez Carlos Sanchez
            rzc Ramzi C
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: