Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60801

Variable changes are shared between matrix (parallel) stages.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: pipeline
    • Labels:
    • Environment:
      Jenkins ver. 2.190.2
      Pipeline: API 2.83
      Pipeline: Basic Steps 2.19
      Pipeline: Declarative 1.5.0
    • Similar Issues:

      Description

      New matrix step provides ability to run the same code in parallel cells. But if that code assigns ANY variable (not just env), it gets overwritten later by the next cell. This is huge deal, it basically prevents from assigning any variable in this kind of pipeline which makes it nearly useless.

      The condition is to set variable, have a piece of code that takes some time to execute and use the already set variable after. This can be simulated with sleep().

      I boiled down my code to easily reproduce it, here is the pipeline:

      pipeline {
          parameters {
              booleanParam(name: 'RHEL5', defaultValue: true)
              booleanParam(name: 'RHEL6', defaultValue: true)
              booleanParam(name: 'RHEL7', defaultValue: true)
          }
          stages {
              stage('Matrix') {
                  matrix {
                      when {
                          anyOf {
                              expression { params.RHEL5 && env.RHEL == '5' }
                              expression { params.RHEL6 && env.RHEL == '6' }
                              expression { params.RHEL7 && env.RHEL == '7' }
                          }
                      }
                      axes {
                          axis {
                              name 'RHEL'
                              values '5', '6', '7'
                          }
                      }
                      stages {
                          stage('Build single'){
                              steps {
                                  script {
                                      echo "RHEL version: " + env.RHEL
                                      myVar = "something.something.RHEL-${env.RHEL}"
                                      echo "myVar: " + myVar
                                      sleep(10)
                                      echo "AFTER RHEL version: " + env.RHEL 
                                      echo "AFTER myVar: " + myVar
                                  }
                              }
                          }
                      }
                  }
              }
          }
      }
      

      The result of this code:

       

      RHEL version: 5
      myVar: something.something.RHEL-5
      Sleeping for 10 sec
      RHEL version: 6
      myVar: something.something.RHEL-6
      Sleeping for 10 sec
      RHEL version: 7
      myVar: something.something.RHEL-7
      Sleeping for 10 sec
      No need to sleep any longer
      RHEL version: 5
      myVar: something.something.RHEL-7
      No need to sleep any longer
      RHEL version: 6
      myVar: something.something.RHEL-7
      No need to sleep any longer
      RHEL version: 7
      myVar: something.something.RHEL-7

      As you can see, myVar gets overwritten by the last run cell even for the other cells.

       

       

        Attachments

          Activity

          Hide
          kryslj Jakub Krysl added a comment - - edited

          Here is a script boiled down to simplest reproducer for both Matrix and Parallel pipeline:

          def getTasks() {
              def tasks = [:]
              //['5', '6', '7'].each { RHEL ->
              versions = ['5', '6', '7']
              for (int i=0; i<versions.size(); i++) {
                  def RHEL = versions[i]
                  tasks["${RHEL}"] = {
                      node('slave') {
                          echo "in PARALLEL"
                          TEST = "in_variable_" + RHEL
                          echo "RHEL: " + RHEL + ", TEST: " + TEST
                          sleep(5)
                          echo "RHEL: " + RHEL + ", TEST: " + TEST
                      }
                  }
              }
              return tasks
          }
          
          pipeline {
              agent { label 'slave' }
              stages {
                  stage('Build Matrix') {
                      matrix {
                          axes {
                              axis {
                                  name 'RHEL'
                                  values '5', '6', '7'
                              }
                          }
                          stages('RHEL') {
                              stage('Build single'){
                                  steps {
                                      script {
                                          echo "in MATRIX"
                                          TEST = "in_variable_" + RHEL
                                          echo "RHEL: " + RHEL + ", TEST: " + TEST
                                          sleep(5)
                                          echo "RHEL: " + RHEL + ", TEST: " + TEST
                                      }
                                  }
                              }
                          }
                      }
                  }
                  stage('Parallel build') {
                      steps {
                          script {
                              parallel getTasks()
                          }
                      }
                  }
              }
          }
          
          

          Output:

          in MATRIX
          RHEL: 5, TEST: in_variable_5
          Sleeping for 5 sec
          in MATRIX
          RHEL: 6, TEST: in_variable_6
          Sleeping for 5 sec
          in MATRIX
          RHEL: 7, TEST: in_variable_7
          Sleeping for 5 sec
          RHEL: 5, TEST: in_variable_7
          RHEL: 6, TEST: in_variable_7
          RHEL: 7, TEST: in_variable_7
          
          Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE
          Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE@2
          Running on jenkins-slave1 in /home/jenkins/workspace/TEST_PIPELINE@2
          in PARALLEL
          RHEL: 5, TEST: in_variable_5
          Sleeping for 5 sec
          in PARALLEL
          RHEL: 6, TEST: in_variable_6
          Sleeping for 5 sec
          in PARALLEL
          RHEL: 7, TEST: in_variable_7
          Sleeping for 5 sec
          RHEL: 5, TEST: in_variable_7
          RHEL: 6, TEST: in_variable_7
          RHEL: 7, TEST: in_variable_7
          Finished: SUCCESS
          

          Setting this issue to blocker as it prevents me from using any cell/branch specific variables.

          Show
          kryslj Jakub Krysl added a comment - - edited Here is a script boiled down to simplest reproducer for both Matrix and Parallel pipeline: def getTasks() { def tasks = [:] //[ '5' , '6' , '7' ].each { RHEL -> versions = [ '5' , '6' , '7' ] for ( int i=0; i<versions.size(); i++) { def RHEL = versions[i] tasks[ "${RHEL}" ] = { node( 'slave' ) { echo "in PARALLEL" TEST = "in_variable_" + RHEL echo "RHEL: " + RHEL + ", TEST: " + TEST sleep(5) echo "RHEL: " + RHEL + ", TEST: " + TEST } } } return tasks } pipeline { agent { label 'slave' } stages { stage( 'Build Matrix' ) { matrix { axes { axis { name 'RHEL' values '5' , '6' , '7' } } stages( 'RHEL' ) { stage( 'Build single' ){ steps { script { echo "in MATRIX" TEST = "in_variable_" + RHEL echo "RHEL: " + RHEL + ", TEST: " + TEST sleep(5) echo "RHEL: " + RHEL + ", TEST: " + TEST } } } } } } stage( 'Parallel build' ) { steps { script { parallel getTasks() } } } } } Output: in MATRIX RHEL: 5, TEST: in_variable_5 Sleeping for 5 sec in MATRIX RHEL: 6, TEST: in_variable_6 Sleeping for 5 sec in MATRIX RHEL: 7, TEST: in_variable_7 Sleeping for 5 sec RHEL: 5, TEST: in_variable_7 RHEL: 6, TEST: in_variable_7 RHEL: 7, TEST: in_variable_7 Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE Running on jenkins-slave2 in /home/jenkins/workspace/TEST_PIPELINE@2 Running on jenkins-slave1 in /home/jenkins/workspace/TEST_PIPELINE@2 in PARALLEL RHEL: 5, TEST: in_variable_5 Sleeping for 5 sec in PARALLEL RHEL: 6, TEST: in_variable_6 Sleeping for 5 sec in PARALLEL RHEL: 7, TEST: in_variable_7 Sleeping for 5 sec RHEL: 5, TEST: in_variable_7 RHEL: 6, TEST: in_variable_7 RHEL: 7, TEST: in_variable_7 Finished: SUCCESS Setting this issue to blocker as it prevents me from using any cell/branch specific variables.
          Hide
          cwoodcox Corey Woodcox added a comment -

          We also see this in a similar build script. We have a job that loops through a list of versions and fires off several docker image builds in parallel and at the end of the script the image name variables have overwritten each other.

          I can try and grab more information if you think it would help troubleshoot.

          Show
          cwoodcox Corey Woodcox added a comment - We also see this in a similar build script. We have a job that loops through a list of versions and fires off several docker image builds in parallel and at the end of the script the image name variables have overwritten each other. I can try and grab more information if you think it would help troubleshoot.
          Hide
          pfuntner John Pfuntner added a comment - - edited

          I'm also seeing this problem and am disappointed that there is no movement on the issue.

          I'm having good luck sticking a def in front of the variable assignment to limit the scope of the variable.  Does this help anyone else?

          Show
          pfuntner John Pfuntner added a comment - - edited I'm also seeing this problem and am disappointed that there is no movement on the issue. I'm having good luck sticking a def in front of the variable assignment to limit the scope of the variable.  Does this help anyone else?
          Hide
          anentropic Anentropic added a comment - - edited

          Same here, this is pretty awful

          John Pfuntner what's an example using `def` that avoids this issue?

          I have tried it in my pipeline but it doesn't seem to help

          Here is another workaround that is useful in some cases (avoids setting a var with cell-specific value) https://stackoverflow.com/a/60258991/202168

          Show
          anentropic Anentropic added a comment - - edited Same here, this is pretty awful John Pfuntner what's an example using `def` that avoids this issue? I have tried it in my pipeline but it doesn't seem to help Here is another workaround that is useful in some cases (avoids setting a var with cell-specific value) https://stackoverflow.com/a/60258991/202168
          Hide
          pfuntner John Pfuntner added a comment -

          Anentropic I'm sorry but I don't think I kept a good example around. I think the script I was using has advanced past that issue.

          Show
          pfuntner John Pfuntner added a comment - Anentropic I'm sorry but I don't think I kept a good example around. I think the script I was using has advanced past that issue.
          Hide
          tonyaps Tony Sinclair added a comment -

          Also hitting this, but not with "matrix"

           

          We're seeing this with "parallel" syntax, but I'm guessing the underlying bug is the same.

           

          I run 3 parallel stages for 3 different architectures.  Currently seeing my "x86_64" arch values suddenly become "ppc64le" arch values.  Builds often fail because they try to run the wrong arch type. 

          Show
          tonyaps Tony Sinclair added a comment - Also hitting this, but not with "matrix"   We're seeing this with "parallel" syntax, but I'm guessing the underlying bug is the same.   I run 3 parallel stages for 3 different architectures.  Currently seeing my "x86_64" arch values suddenly become "ppc64le" arch values.  Builds often fail because they try to run the wrong arch type. 
          Hide
          benbrummer Benjamin Brummer added a comment -

          Without "def toolName" i can reproduce this issue. This is a post-stage inside a matrix pipeline job.

          post {
              always {
          ...
                  script {
                      // Without def toolName the variable scope is for the whole pipeline
                      // and therefore in a Matrix/Parallel build we have stages overwriting
                      // each other
                      def toolName
                      switch ("${TOOL}") {
                          case 'configurator':
                              toolName = 'Kyoto Configurator'
                              break
                          case 'designer':
                              toolName = 'Kyoto Designer'
                              break
                      }
          ...
              }
          }
          
          Show
          benbrummer Benjamin Brummer added a comment - Without "def toolName" i can reproduce this issue. This is a post-stage inside a matrix pipeline job. post { always { ... script { // Without def toolName the variable scope is for the whole pipeline // and therefore in a Matrix/Parallel build we have stages overwriting // each other def toolName switch ( "${TOOL}" ) { case 'configurator' : toolName = 'Kyoto Configurator' break case 'designer' : toolName = 'Kyoto Designer' break } ... } }

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            kryslj Jakub Krysl
            Votes:
            9 Vote for this issue
            Watchers:
            11 Start watching this issue

              Dates

              Created:
              Updated: