Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46831

Not able to use a normal agent on stage when agent docker is declared globally

      When the agent globally defined is a a docker-based, then if a stage defined another agent, it seems that we are facing JENKINS-33510.

       

      PR: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/192

          [JENKINS-46831] Not able to use a normal agent on stage when agent docker is declared globally

          What's the state of this issue ?

          When will it be possible to run stages in parallel with docker agents ?

          wknapik wknapik added a comment - What's the state of this issue ? When will it be possible to run stages in parallel with docker agents ?

          This issue, marked as critical, has been open for 6 months with no apparent progress.

          Making it impossible for users to run stages in parallel goes against the core purpose of this software, which is providing rapid feedback.

          It's as if `ls' stopped listing some files and for half a year no one got around to fixing it.

          abayer - can you please comment ? When can we expect this to be resolved ? How can users, who have built their pipelines using docker-based agents, run stages in parallel ? Is there some workaround/hack that can be used now ?

          wknapik wknapik added a comment - This issue, marked as critical, has been open for 6 months with no apparent progress. Making it impossible for users to run stages in parallel goes against the core purpose of this software, which is providing rapid feedback. It's as if `ls' stopped listing some files and for half a year no one got around to fixing it. abayer - can you please comment ? When can we expect this to be resolved ? How can users, who have built their pipelines using docker-based agents, run stages in parallel ? Is there some workaround/hack that can be used now ?

          abayer ping?

          wknapik wknapik added a comment - abayer ping?

          Andrew Bayer added a comment -

          It's due to an inherent aspect of the Docker Pipeline plugin - we're working on a successor to Docker Pipeline over in JENKINS-48050, but I don't have a specific timeframe. For now, you just have to not use a top-level docker agent if you want to run stages, parallel or otherwise, on a different agent.

          Andrew Bayer added a comment - It's due to an inherent aspect of the Docker Pipeline plugin - we're working on a successor to Docker Pipeline over in JENKINS-48050 , but I don't have a specific timeframe. For now, you just have to not use a top-level docker agent if you want to run stages, parallel or otherwise, on a different agent.

          abayer thanks for the response. I don't really care where/how the agents are specified - I just need to run all the stages in docker containers. Are you saying that it's possible to run parallel stages in containers (on multiple slaves at the same time), if I change how the agents are specified in the Jenkinsfile ?

          wknapik wknapik added a comment - abayer thanks for the response. I don't really care where/how the agents are specified - I just need to run all the stages in docker containers. Are you saying that it's possible to run parallel stages in containers (on multiple slaves at the same time), if I change how the agents are specified in the Jenkinsfile ?

          abayer I tried doing that (as described in JENKINS-47103), but it didn't work back then. Has something changed ?

          wknapik wknapik added a comment - abayer I tried doing that (as described in  JENKINS-47103 ), but it didn't work back then. Has something changed ?

          Martin Sander added a comment - - edited

          abayer:

          I have a problem that might be this one or another issue:

          pipeline {
              agent {
                  docker {
                      image 'microsoft/azure-cli:2.0.28'
                      args '-u 0:0'
                  }
              }
              options { timestamps() }
          
              stages {
                  stage('azure-cli') {
                      steps {
                          sh 'az --version'
                      }
                  }
                  stage('dotnet') {
                      agent {
                          docker {
                              image 'microsoft/dotnet:2.0-runtime-stretch'
                              args '-u 0:0'
                          }
                      }
                      steps {
                          sh 'ls /usr/share/dotnet'
                      }
                  }
              }
          }
          

          stage('azure-cli') works, stage('dotnet') fails with: Error: No such container: c0ffeec0de (which is actually the container that runs microsoft/azure-cli:2.0.28, but is running on a completely different node).

          Using reuseNode for the dotnet docker agent leads to a different error: Jenkins tries to start the second docker container from within the first one.

          Is this the same issue as this or a new one? Plugin version 1.2.7.

          Martin Sander added a comment - - edited abayer : I have a problem that might be this one or another issue: pipeline { agent { docker { image 'microsoft/azure-cli:2.0.28' args '-u 0:0' } } options { timestamps() } stages { stage( 'azure-cli' ) { steps { sh 'az --version' } } stage( 'dotnet' ) { agent { docker { image 'microsoft/dotnet:2.0-runtime-stretch' args '-u 0:0' } } steps { sh 'ls /usr/share/dotnet' } } } } stage('azure-cli') works, stage('dotnet') fails with: Error: No such container: c0ffeec0de (which is actually the container that runs microsoft/azure-cli:2.0.28 , but is running on a completely different node). Using reuseNode for the dotnet docker agent leads to a different error: Jenkins tries to start the second docker container from within the first one. Is this the same issue as this or a new one? Plugin version 1.2.7.

          Andrew Bayer added a comment -

          0x89 - Yeah, it's the same issue, basically. What I'd recommend is that you do agent any at the top level, then docker agents for each stage with reuseNode true - should get you the behavior you're looking for.

          Andrew Bayer added a comment - 0x89 - Yeah, it's the same issue, basically. What I'd recommend is that you do agent any at the top level, then docker agents for each stage with reuseNode true - should get you the behavior you're looking for.

          trejkaz added a comment -

          We hit this one today, but the issue linked in the description (JENKINS-33510) is marked as resolved. I assume it's now a different issue? I get the same error as above ("No such container: ------").

          trejkaz added a comment - We hit this one today, but the issue linked in the description ( JENKINS-33510 ) is marked as resolved. I assume it's now a different issue? I get the same error as above ("No such container: ------").

          the mentioned pr was merged
          but the introduced test was removed again https://github.com/jenkinsci/pipeline-model-definition-plugin/commit/47504a0b4507b1908472fe98fd44d9859a447abd by jglick
          does that mean that this will not get addressed at all?

          Kevin Broselge added a comment - the mentioned pr was merged but the introduced test was removed again https://github.com/jenkinsci/pipeline-model-definition-plugin/commit/47504a0b4507b1908472fe98fd44d9859a447abd by jglick does that mean that this will not get addressed at all?

            Unassigned Unassigned
            alecharp Adrien Lecharpentier
            Votes:
            9 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: