Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40986

Matrix structure for Declarative Pipeline

    XMLWordPrintable

    Details

    • Similar Issues:
    • Epic Link:
    • Sprint:
      Declarative backlog

      Attachments

        Issue Links

          Activity

          arthurlutz Arthur Lutz created issue -
          jamesdumay James Dumay made changes -
          Field Original Value New Value
          Component/s pipeline-model-definition-plugin [ 21706 ]
          jamesdumay James Dumay made changes -
          Component/s blueocean-plugin [ 21481 ]
          Hide
          jamesdumay James Dumay added a comment -

          Arthur Lutz would something like this solve the matrix capability you are looking for?

          pipeline {
            stages {
              stage(‘browser tests’) {
                matrix {
                  variants {
                    variant (‘firefox’) {
                      agent { node “linux+firefox” }
                	    environment { key: value }
                    }
                    variant ('chrome') {
                      agent { node 'linux+chrome' }
                	    environment { key: value }
                    }
                	}
                  environment { … }
                  stages { … }
                }
              }
          }
          
          Show
          jamesdumay James Dumay added a comment - Arthur Lutz would something like this solve the matrix capability you are looking for? pipeline { stages { stage(‘browser tests’) { matrix { variants { variant (‘firefox’) { agent { node “linux+firefox” } environment { key: value } } variant ( 'chrome' ) { agent { node 'linux+chrome' } environment { key: value } } } environment { … } stages { … } } } }
          abayer Andrew Bayer made changes -
          Assignee Andrew Bayer [ abayer ]
          abayer Andrew Bayer made changes -
          Link This issue relates to JENKINS-41334 [ JENKINS-41334 ]
          abayer Andrew Bayer made changes -
          Link This issue relates to JENKINS-27395 [ JENKINS-27395 ]
          jamesdumay James Dumay made changes -
          Epic Link JENKINS-45425 [ 183593 ]
          jamesdumay James Dumay made changes -
          Summary Extend Declarative Pipeline so that it has the matrix capability Matrix structure for Declarative Pipeline
          abayer Andrew Bayer made changes -
          Link This issue relates to JENKINS-46809 [ JENKINS-46809 ]
          Hide
          robhales Robert Hales added a comment -

          I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy. 

           

           

          https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227

           

          Show
          robhales Robert Hales added a comment - I was just pointed at this JIRA today. The syntax above would be extremely wordy for any by the simplest combination. Even something with 4 OSs and 3 variants would be 12 blocks that would have to be written.  This would be much easier to work with if you could provide a map of lists or list of lists and recursively iterate over them to come up with all the combinations. I have done this in pipeline using groovy code and it works well this way. An official syntax might be handy.      https://stackoverflow.com/questions/46601999/jenkins-pipeline-multiconfiguration-project/46675227#46675227  
          Hide
          abayer Andrew Bayer added a comment -

          Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809, so that part's done. That leaves the axis definitions.

          First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types":

          • name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional.
          • agent - which would be 1..n possible agent block contents.
          • environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix, but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group, so if you've got common env vars across all the permutations, that's where you'd define them.
          • tools - like agent, 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis.

          I am nowhere near thinking of the syntax yet, and this is still months away from actually happening (JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.

          Show
          abayer Andrew Bayer added a comment - Starting to think this one through now that JENKINS-46809 is churning along. The body will basically be a group as will be added by JENKINS-46809 , so that part's done. That leaves the axis definitions. First, I'm thinking we do ape Matrix jobs - not defining a full "variant" but individual axes and their possible values, leaving the behind the scenes magic to come up with the combinations. My initial thoughts are that we have the following axis "types": name - a base name that we make unique by combining the other axes somehow. Optional, probably. All the others are definitely optional. agent - which would be 1..n possible agent block contents. environmentVariable - better name still to come. =) This would be a single env var and its 1..n possible values. You can specify more than one of these per matrix , but again thinking of how Matrix jobs work, we don't want to just have a big ol' block of multiple variables that you have to copy and tweak a few times to get all the combinations you want. The current thinking on JENKINS-46809 will let you specify an environment block for a stage group , so if you've got common env vars across all the permutations, that's where you'd define them. tools - like agent , 1..n possible tools block contents. Again, similar to Matrix jobs letting you use JDK or Maven version as an axis. I am nowhere near thinking of the syntax yet, and this is still months away from actually happening ( JENKINS-46809 has to be completed, Blue Ocean and the editor have to be updated accordingly, and then we can implement matrix in Declarative followed by its Blue Ocean/editor changes), but I definitely see how we'll get there now implementation-wise. Woo.
          abayer Andrew Bayer made changes -
          Link This issue is duplicated by JENKINS-49662 [ JENKINS-49662 ]
          Hide
          neothethird Jan Sprinz added a comment -

          Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines.

          I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...

          Show
          neothethird Jan Sprinz added a comment - Any news on this? Seems like most of the prerequisites are out of the way and this would be an absolute killer feature to have for declarative pipelines. I know it's way to early to think about visualization, but i'm not sure it's a good idea to include this in the normal blueocean pipeline view, since it would get really cluttered really fast. Maybe there could be a popup presenting something similar to the visualization of matrix builds in classic jenkins? I have a job with seven dimensions, i won't even try to imagine what it would look like to have all of those displayed in the normal pipeline view...
          abayer Andrew Bayer made changes -
          Sprint Declarative backlog [ 621 ]
          Hide
          seckler Steffen Seckler added a comment -

          could we up the priority of this?
          seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...

          Show
          seckler Steffen Seckler added a comment - could we up the priority of this? seeing that most major other CI tools have this kind of feature, it should be implemented in jenkins rather soon...
          Hide
          neothethird Jan Sprinz added a comment -

          Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.

          Show
          neothethird Jan Sprinz added a comment - Yes please! It's a must-have, really. The workaround of using loops in a scripted block for this is ugly and error-prone... All i really want for Christmas Easter is declarative Jenkins matrices.
          neothethird Jan Sprinz made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          skechav Sakis Kechlimparis made changes -
          Rank Ranked higher
          Hide
          trejkaz trejkaz added a comment -

          I drafted up what I would hope this sort of thing would look like, and ended up with something like:

          pipeline {
              // ...
              stages {
                  // ...
                  matrixBuild {
                      matrix [label: 'debian',  prettyName: 'Debian'],
                             [label: 'ubuntu',  prettyName: 'Ubuntu'],
                             [label: 'centos',  prettyName: 'CentOS'],
                             [label: 'macos',   prettyName: 'macOS'],
                             [label: 'windows', prettyName: 'Windows']
          
                      template {
                          stage("Test on ${prettyName}") {
                              agent {
                                  label name
                              }
                              steps {
                                  unstash 'compile-artifacts'
                                  unstash 'dot-gradle'
          
                                  gradle tasks: 'check', switches: '--stacktrace'
                              }
                              post {
                                  always {
                                      junit '*/build/test-results/**/*.xml'
                                  }
                              }
                          }
                      }
                  }
                  // ...
              }
          }
          

          The values in the maps for matrix would just become available in the context for template, and if you wanted to put whole closures in as the values, that's fine too.

          Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.

          Show
          trejkaz trejkaz added a comment - I drafted up what I would hope this sort of thing would look like, and ended up with something like: pipeline { // ... stages { // ... matrixBuild { matrix [label: 'debian' , prettyName: 'Debian' ], [label: 'ubuntu' , prettyName: 'Ubuntu' ], [label: 'centos' , prettyName: 'CentOS' ], [label: 'macos' , prettyName: 'macOS' ], [label: 'windows' , prettyName: 'Windows' ] template { stage( "Test on ${prettyName}" ) { agent { label name } steps { unstash 'compile-artifacts' unstash 'dot-gradle' gradle tasks: 'check' , switches: '--stacktrace' } post { always { junit '*/build/test-results /**/ *.xml' } } } } } // ... } } The values in the maps for matrix would just become available in the context for template , and if you wanted to put whole closures in as the values, that's fine too. Practicality, though, no idea at all. I was going to try to do it as a pipeline library some day if free time ever came up.
          bitwiseman Liam Newman made changes -
          Assignee Andrew Bayer [ abayer ] Liam Newman [ bitwiseman ]
          bitwiseman Liam Newman made changes -
          Remote Link This issue links to "PR-251 (Web Link)" [ 23024 ]
          Hide
          maydjin Denis Dudarev added a comment - - edited

          It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.

           

          User case:

           * User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
           * User has combination of platforms ('win32', 'linux', 'osx', 'android')
           * User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
           * User desired that combination '.*'-'linux|osx|android'-'cl' doesn't make sense
           * User desired that combination 'arm.*'-'win32|osx'-'.*' doesn't make sense
           * User desired that combination '.*'-'osx'-'gcc' is not supported yet

           

          It could be regexp syntax, as I offered before, or more imperative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.

           

          Also, there should be a way to skip stage/step on given axises combination.

          User case:

          * User has combination of architectures ('x86',  'x86_64', 'armv7', 'armv8')
          * User has combination of platforms ('win32', 'linux', 'osx', 'android')
          * User has combination of compilers ('clang', 'gcc', 'cl', 'icc')
          * User has stage 'build'
          * User has stage 'build MSI package'
          * User desired that 'build' stage makes sense in any combination of axises
          * User desired that 'build MSI package' for '.*'-'linux|osx|android'-'.*' doesn't make sense
          

          Andrew Bayer, please let me know, if there is any design draft of upcoming API.

          Show
          maydjin Denis Dudarev added a comment - - edited It would be great if not only 2d matrices would be supported(more than 2 axis should be supported). As well only sparsable matrices would be useful.   User case: * User has combination of architectures ( 'x86' , 'x86_64' , 'armv7' , 'armv8' ) * User has combination of platforms ( 'win32' , 'linux' , 'osx' , 'android' ) * User has combination of compilers ( 'clang' , 'gcc' , 'cl' , 'icc' ) * User desired that combination '.*' - 'linux|osx|android' - 'cl' doesn't make sense * User desired that combination 'arm.*' - 'win32|osx' - '.*' doesn't make sense * User desired that combination '.*' - 'osx' - 'gcc' is not supported yet   It could be regexp syntax, as I offered before, or more imperative like in original "Matrix job". IMHO, only this approach could make sense in 90% user cases.   Also, there should be a way to skip stage/step on given axises combination. User case: * User has combination of architectures ( 'x86' , 'x86_64' , 'armv7' , 'armv8' ) * User has combination of platforms ( 'win32' , 'linux' , 'osx' , 'android' ) * User has combination of compilers ( 'clang' , 'gcc' , 'cl' , 'icc' ) * User has stage 'build' * User has stage 'build MSI package' * User desired that 'build' stage makes sense in any combination of axises * User desired that 'build MSI package' for '.*' - 'linux|osx|android' - '.*' doesn't make sense Andrew Bayer , please let me know, if there is any design draft of upcoming API.
          zbynek Zbynek Konecny made changes -
          Link This issue is duplicated by JENKINS-42808 [ JENKINS-42808 ]
          zbynek Zbynek Konecny made changes -
          Link This issue is duplicated by JENKINS-42808 [ JENKINS-42808 ]
          zbynek Zbynek Konecny made changes -
          Link This issue relates to JENKINS-42808 [ JENKINS-42808 ]
          Hide
          bitwiseman Liam Newman added a comment -

          You can pull 1.5.0-beta1 from the experimental update site now to take a look at matrix.

          Show
          bitwiseman Liam Newman added a comment - You can pull 1.5.0-beta1 from the experimental update site now to take a look at matrix.
          Hide
          sshepel Serhii Shepel added a comment -

          Liam Newman could you please provide some info regarding usage and possibilities for the bindings...

          I can see a bunch of test cases here: (https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/335e44ff6d19cfd43f58be210d9af81e4491cdec/pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/MatrixTest.java), but a bit more details would be nice...

          Show
          sshepel Serhii Shepel added a comment - Liam Newman could you please provide some info regarding usage and possibilities for the bindings... I can see a bunch of test cases here: ( https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/335e44ff6d19cfd43f58be210d9af81e4491cdec/pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/MatrixTest.java ), but a bit more details would be nice...
          Hide
          seckler Steffen Seckler added a comment -
          Show
          seckler Steffen Seckler added a comment - from what I see, the files in the json test directory can give very good hints: https://github.com/jenkinsci/pipeline-model-definition-plugin/tree/master/pipeline-model-definition/src/test/resources/json/matrix
          Hide
          sshepel Serhii Shepel added a comment -

          Ah right!
          Thanks Steffen Seckler

          Show
          sshepel Serhii Shepel added a comment - Ah right! Thanks Steffen Seckler
          Hide
          maydjin Denis Dudarev added a comment -

          Thank you for great feature!

          I could only see 2 axis cases in tests and json files. Is this a limit for there?

          Anyway I need better instructions to check it out. For example - sample Jenkinsfile with 3axis build would be very useful.

          Show
          maydjin Denis Dudarev added a comment - Thank you for great feature! I could only see 2 axis cases in tests and json files. Is this a limit for there? Anyway I need better instructions to check it out. For example - sample Jenkinsfile with 3axis build would be very useful.
          Hide
          maydjin Denis Dudarev added a comment -

          Hm. I've reviewed json file more detailed. It seems to be quite agile implementation, covering most of cases above. So what I need (and most users I believe) is a Jenkinsfile example, to review end-user syntax.

          Show
          maydjin Denis Dudarev added a comment - Hm. I've reviewed json file more detailed. It seems to be quite agile implementation, covering most of cases above. So what I need (and most users I believe) is a Jenkinsfile example, to review end-user syntax.
          Show
          abayer Andrew Bayer added a comment - https://github.com/jenkinsci/pipeline-model-definition-plugin/tree/master/pipeline-model-definition/src/test/resources/matrix has a whole bunch of Jenkinsfile examples/test cases. =)
          Hide
          maydjin Denis Dudarev added a comment -

          Got it, thanks!

          Show
          maydjin Denis Dudarev added a comment - Got it, thanks!
          Hide
          maydjin Denis Dudarev added a comment -

          Results of my humble review:

          • It seems to be usable, it covers user cases
          • Excludes syntax is quite huge, but I have no fast good idea for much compact one without imperative expressions from original matrix functionality, it could be very annoying to avoid all uninterested combinations with axis count >2, especially in cases where *most* of combinations are uninterested (e.g. at the start of a project)
          Show
          maydjin Denis Dudarev added a comment - Results of my humble review: It seems to be usable, it covers user cases Excludes syntax is quite huge, but I have no fast good idea for much compact one without imperative expressions from original matrix functionality, it could be very annoying to avoid all uninterested combinations with axis count >2, especially in cases where * most * of combinations are uninterested (e.g. at the start of a project)
          pandastic Alexandre Fabra Fernandez made changes -
          Rank Ranked lower
          Hide
          simon_lg Simon Larouche-Gagne added a comment -

          +1 on the exclude syntax, it's taking a lot of space.
          Also, can the values field of the axis block be dynamic instead of using a filter parameter?
          I have a use case with axes with a lot of values depending on a parameter, so if I can't act on the matrix values, my only option is to handle it in a when block?
          Also, in most cases, I would rather use a global variable to store my axes values as well, makes it easier to maintain and I could re-use it in another matrix block further down in the script.
          But overall I'm very happy to see a matrix block in declarative pipeline.

          Show
          simon_lg Simon Larouche-Gagne added a comment - +1 on the exclude syntax, it's taking a lot of space. Also, can the values field of the axis block be dynamic instead of using a filter parameter? I have a use case with axes with a lot of values depending on a parameter, so if I can't act on the matrix values, my only option is to handle it in a when block? Also, in most cases, I would rather use a global variable to store my axes values as well, makes it easier to maintain and I could re-use it in another matrix block further down in the script. But overall I'm very happy to see a matrix block in declarative pipeline.
          neothethird Jan Sprinz made changes -
          Link This issue relates to JENKINS-56000 [ JENKINS-56000 ]
          Hide
          wohali Joan Touzet added a comment -

          One enhancement that might be nice, if possible, is to have the values for an axis name be supported as non-literals. I don't seem to be able to make this work in the current version. Sorry if this isn't the right place, but you were asking for examples above...

           

          Example:

          pipeline {
            environment {
               LOW_ERLANG_VER = '19.3.6.8'
               MID_ERLANG_VER = '20.3.8.24'
               HIGH_ERLANG_VER = '22.2'
          {{  }}}

            stages {
              // ...
              // Other stages that need the values above
              // ...
              stage("Build") {
                matrix {
                  axes {
                    axis {
                      name 'ERLANG_VERSION'
                      values "{$env.LOW_ERLANG_VER}", "${env.MID_ERLANG_VER}", "${env.HIGH_ERLANG_VER}"
          {{          }}}
                  } // axes
                  // ...
                  // do the build with ${ERLANG_VERSION}
                  // ...
                } // matrix
              } // stage
            } // stages
          } // pipeline

          if that makes sense.
          Show
          wohali Joan Touzet added a comment - One enhancement that might be nice, if possible, is to have the values for an axis name be supported as non-literals. I don't seem to be able to make this work in the current version. Sorry if this isn't the right place, but you were asking for examples above...   Example: pipeline {   environment {      LOW_ERLANG_VER = '19.3.6.8'      MID_ERLANG_VER = '20.3.8.24'      HIGH_ERLANG_VER = '22.2' {{  }}}   stages {     // ...     // Other stages that need the values above     // ...     stage("Build") {       matrix {         axes {           axis {             name 'ERLANG_VERSION'             values "{$env.LOW_ERLANG_VER}", "${env.MID_ERLANG_VER}", "${env.HIGH_ERLANG_VER}" {{          }}}         } // axes         // ...         // do the build with ${ERLANG_VERSION }         // ...       } // matrix     } // stage   } // stages } // pipeline if that makes sense.
          Hide
          bitwiseman Liam Newman added a comment -

          Joan Touzet
          This totally makes sense. An issue has been opened for it. https://issues.jenkins-ci.org/browse/JENKINS-61047

          Show
          bitwiseman Liam Newman added a comment - Joan Touzet This totally makes sense. An issue has been opened for it. https://issues.jenkins-ci.org/browse/JENKINS-61047

            People

            Assignee:
            bitwiseman Liam Newman
            Reporter:
            arthurlutz Arthur Lutz
            Votes:
            32 Vote for this issue
            Watchers:
            41 Start watching this issue

              Dates

              Created:
              Updated: