Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64949

Declarative Pipeline: Allow defining matrix axes from a file or a variable

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      As a Jenkins user writing pipelines from my projects,
      When I use a matrix in Declarative syntax because this syntax is really efficient and easy to read and maintain
      Then I would want to allow defining dynamically some axis from a file content and/or a variable defined outside the scope of the pipeline block

      Examples of the expectations (that are only suggestions):

      pipeline {
        // ..
        stages {
          stage("Build") {
            matrix {
              axes {
                axis {
                  name 'GOARCH'
                  values readYaml file: 'archs.yml'
                }
      // ...
      

      with the following file:

      # archs.yml
      - amd64
      - i368
      - s390x
      
      //...
                axis {
                  name 'GOARCH'
                  values readJson file: 'archs.json'
                }
      //...
      
      ["amd64", "i386","s390x"]
      
      • Eventually with a dumb text file with 1 value per line:
      //...
                axis {
                  name 'GOARCH'
                  values readTxt file: 'archs.txt
                }
      //...
      
      amd64
      i386
      s390x
      
      • Or from a variable defined outside the `pipeline {}` block:
      def architectureList = readYaml file: "archs.yaml"
      
      pipeline {
      //...
         axis { 
           name 'GOARCH' 
           values architectureList
        }
      

      Of course, the scripted syntax solves this issue, but the cost of switching syntax is high (not mentioning that it does not fit all the pipeline writers).

      Inspiration come from https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/#new-fromjson-method-in-expressions.

      It's interesting how CDS and Gitlab are "solving" the dynamic matrix issue by using "templates" that generate sub pipelines for each matrix axe: https://ovh.github.io/cds/docs/concepts/template/ and https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/45795/diffs.

        Attachments

          Activity

          dduportal Damien Duportal created issue -
          Hide
          dduportal Damien Duportal added a comment -

          I understand that it might be related to JENKINS-61047, JENKINS-61504 and JENKINS-61280

          Show
          dduportal Damien Duportal added a comment - I understand that it might be related to JENKINS-61047 , JENKINS-61504 and JENKINS-61280
          dduportal Damien Duportal made changes -
          Field Original Value New Value
          Description As a Jenkins user writing pipelines from my projects,
          When I use a matrix in Declarative syntax because this syntax is really efficient and easy to read and maintain
          Then I would want to allow defining dynamically some axis from a file content and/or a variable defined outside the scope of the pipeline block

          Examples of the expectations (that are only suggestions):

          * With a [readYaml helper method|https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readyaml-read-yaml-from-files-in-the-workspace-or-text]:

          {code}
          pipeline {
            // ..
            stages {
              stage("Build") {
                matrix {
                  axes {
                    axis {
                      name 'GOARCH'
                      values readYaml file: 'archs.yml'
                    }
          // ...
          {code}

          with the following file:

          {code}
          # archs.yml
          - amd64
          - i368
          - s390x
          {code}

          * Same idea with a JSON file:
          {code}
          //...
                    axis {
                      name 'GOARCH'
                      values readJson file: 'archs.json'
                    }
          //...
          {code}

          {code}
          ["amd64", "i386","s390x"]
          {code}

          * Eventually with a dumb text file with 1 value per line:
          {code}
          //...
                    axis {
                      name 'GOARCH'
                      values readTxt file: 'archs.txt
                    }
          //...
          {code}

          {code}
          amd64
          i386
          s390x
          {code}

          * Or from a variable defined outside the `pipeline {}` block:

          {code}
          def architectureList = readYaml file: "archs.yaml"

          pipeline {
          // ...
          {code}
          //...
                    axis {
                      name 'GOARCH'
                      values architectureList
                    }
          //...
          {code}

          Of course, the scripted syntax solves this issue, but the cost of switching syntax is high (not mentioning that it does not fit all the pipeline writers).

          Inspiration come from https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/#new-fromjson-method-in-expressions.

          It's interesting how CDS and Gitlab are "solving" the dynamic matrix issue by using "templates" that generate sub pipelines for each matrix axe: https://ovh.github.io/cds/docs/concepts/template/ and https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/45795/diffs.


          As a Jenkins user writing pipelines from my projects,
           When I use a matrix in Declarative syntax because this syntax is really efficient and easy to read and maintain
           Then I would want to allow defining dynamically some axis from a file content and/or a variable defined outside the scope of the pipeline block

          Examples of the expectations (that are only suggestions):

           * With a [readYaml helper method|https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readyaml-read-yaml-from-files-in-the-workspace-or-text]:

          {code:java}
          pipeline {
            // ..
            stages {
              stage("Build") {
                matrix {
                  axes {
                    axis {
                      name 'GOARCH'
                      values readYaml file: 'archs.yml'
                    }
          // ...
          {code}
          with the following file:
          {code:java}
          # archs.yml
          - amd64
          - i368
          - s390x
          {code}

           * Same idea with a JSON file (and the [readJson helper|https://www.jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace]):

          {code:java}
          //...
                    axis {
                      name 'GOARCH'
                      values readJson file: 'archs.json'
                    }
          //...
          {code}

          {code:java}
          ["amd64", "i386","s390x"]
          {code}

           * Eventually with a dumb text file with 1 value per line:

          {code:java}
          //...
                    axis {
                      name 'GOARCH'
                      values readTxt file: 'archs.txt
                    }
          //...
          {code}

          {code:java}
          amd64
          i386
          s390x
          {code}

           * Or from a variable defined outside the `pipeline {}` block:

          {code:java}
          def architectureList = readYaml file: "archs.yaml"

          pipeline {
          //...
             axis {
               name 'GOARCH'
               values architectureList
            }
          {code}
          Of course, the scripted syntax solves this issue, but the cost of switching syntax is high (not mentioning that it does not fit all the pipeline writers).

          Inspiration come from [https://github.blog/changelog/2020-04-15-github-actions-new-workflow-features/#new-fromjson-method-in-expressions].

          It's interesting how CDS and Gitlab are "solving" the dynamic matrix issue by using "templates" that generate sub pipelines for each matrix axe: [https://ovh.github.io/cds/docs/concepts/template/] and [https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/45795/diffs].
          Hide
          jglick Jesse Glick added a comment -

          CDS and Gitlab are "solving" the dynamic matrix issue by using "templates" that generate sub pipelines for each matrix ax[is]

          Basically what Jenkins did in matrix-project, superseded by Pipeline.

          Show
          jglick Jesse Glick added a comment - CDS and Gitlab are "solving" the dynamic matrix issue by using "templates" that generate sub pipelines for each matrix ax[is] Basically what Jenkins did in matrix-project , superseded by Pipeline.
          Hide
          dduportal Damien Duportal added a comment -

          > Basically what Jenkins did in matrix-project, superseded by Pipeline.

          Yes, but without the "config as code" setup as far as I understand?

          Show
          dduportal Damien Duportal added a comment - > Basically what Jenkins did in matrix-project , superseded by Pipeline. Yes, but without the "config as code" setup as far as I understand?
          Hide
          jglick Jesse Glick added a comment -

          Yes, except to the extent you can use job-dsl for that aspect.

          Show
          jglick Jesse Glick added a comment - Yes, except to the extent you can use job-dsl for that aspect.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            dduportal Damien Duportal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: