Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46809

Allow sequential stages inside parallel in Declarative syntax

    • Pipeline - April 2018

      Currently with -JENKINS-41334-, we set one or more stages to be executed in parallel but no "branches". Meaning that we cannot parallelise a sequence of stages.

       

      Example:

      pipeline {
        agent none
        stages {
          stage('Parallel stuff') {
            parallel 'branch 1' : {
              // Sequencial stages
              stage('Branch 1 stage 1'){
                  agent any
                  steps {
                      echo "In branch 1 stage 1"
                  }
              }
              stage('Branch 1 stage 2'){
                  agent none // With that kind of sequencial stage, we can change the agent to run on
                  steps {
                      sleep 30
                  }
              }
            }, 'branch 2': { // Parallel execution
              stage('Branch 2 stage 1'){
                  agent any
                  steps {
                      echo "In branch 2 stage 1"
                      sleep 60
                  }
              }
            }
          }
        }
      }
      

      Blue ocean possible view:

          [JENKINS-46809] Allow sequential stages inside parallel in Declarative syntax

          Bastien Arata created issue -
          Andrew Bayer made changes -
          Issue Type Original: Bug [ 1 ] New: New Feature [ 2 ]

          Andrew Bayer added a comment -

          So this is on the longer-term roadmap. I'm not sure yet what the syntax will look like, but the necessary tooling in Blue Ocean to support this sort of visualization is beginning to come together. So hopefully we'll be able to get this in a few months.

          Andrew Bayer added a comment - So this is on the longer-term roadmap. I'm not sure yet what the syntax will look like, but the necessary tooling in Blue Ocean to support this sort of visualization is beginning to come together. So hopefully we'll be able to get this in a few months.

          Bastien Arata added a comment - - edited

          Great to hear that ! The syntax doesn't matter (mine is not even syntactically correct), I was just throwing an example to clarify myself. Any issues to watch in order to be aware of any progression on the subject ?

          Bastien Arata added a comment - - edited Great to hear that ! The syntax doesn't matter (mine is not even syntactically correct), I was just throwing an example to clarify myself. Any issues to watch in order to be aware of any progression on the subject ?

          Andrew Bayer added a comment -

          This one, I guess. =)

          Andrew Bayer added a comment - This one, I guess. =)

          Bastien Arata added a comment -

          Ok then =)

          Bastien Arata added a comment - Ok then =)
          Andrew Bayer made changes -
          Link New: This issue relates to JENKINS-40986 [ JENKINS-40986 ]
          Andrew Bayer made changes -
          Link New: This issue is duplicated by JENKINS-47532 [ JENKINS-47532 ]

          Andrew Bayer added a comment - - edited

          Thinking about syntax here - some possibilities:

          First, I think this feels generally right for defining a "group" of stages to run sequentially:

          group('some-name') {
            stage('a') { ... }
            stage('b') { ... }
          }
          

          I'm open to something other than "group". The main thing is to have a name for the group - this ends up, behind the scenes, being the name of the parallel step branch the group's stages get executed within. I decided to go with group('some-name') rather than a nested name 'some-name' field because if we did the latter, we'd either have to add another nested stages or something like that for the individual stage definitions to be in, or we'd have the contents of group being either name ... or stage(...). I don't like mixing key/value pair fields in the same block as arbitrary-lists-of-things, so the latter doesn't work for me, and the former adds more verbosity than I think is really needed.

          Second is how we handle declaring parallel groups in a stage, and related, how we handle existing parallel stages. I've played around with having parallel be able to contain either a list of stage or a list of group - or even a mix. Neither approach (exclusively stage or group, or non-exclusive) feels very smooth in implementation, though the syntax is pretty nice, especially for the mixed approach. I've also considered a new section, let's call it parallelGroups for now, where you could either have parallel or parallelGroups on a stage but not both. Much easier implementation-wise, better backwards compatibility (most notably for the editor, which wouldn't have to change how it handles parallel at all, just add parallelGroups support and make sure only one of the two can be used), worse syntax.

          So here are the options as I see it:

          Exclusive lists in parallel

          stage('just-stages') {
            parallel {
              // Just stages in this case
              stage('a') { ... }
              stage('b') { ... }
              ...
            }
          }
          stage('just-groups') {
            parallel {
              // Just groups in this case - note that we still need a group for just one stage
             group('first-group') {
               stage('a) { ... }
             }
             group('second-group') {
               stage('b') { ... }
               stage('c') { ... }
            }
          }
          

          Mixed lists in parallel

          stage('mixed-list') {
            parallel {
              stage('a') { ... }
              group('b-and-c') {
                stage('b') { ... }
                stage('c') { ... }
              }
            }
          }
          

          parallel and parallelGroups

          stage('parallel-single-stages') {
            parallel {
              stage('a') { ... }
              stage('b') { ... }
            }
          }
          stage('parallelGroups') {
            parallelGroups {
              group('first-group') {
                stage('a') { ... }
              }
              group('second-group') {
                stage('b') { ... }
                stage('c') { ... }
              }
            }
          }
          

          Thoughts? (ping kshultz michaelneale rsandell jamesdumay stephendonner rpocase hrmpw)

          Andrew Bayer added a comment - - edited Thinking about syntax here - some possibilities: First, I think this feels generally right for defining a "group" of stages to run sequentially: group( 'some-name' ) { stage( 'a' ) { ... } stage( 'b' ) { ... } } I'm open to something other than "group". The main thing is to have a name for the group - this ends up, behind the scenes, being the name of the parallel step branch the group's stages get executed within. I decided to go with group('some-name') rather than a nested name 'some-name' field because if we did the latter, we'd either have to add another nested stages or something like that for the individual stage definitions to be in, or we'd have the contents of group being either name ... or stage(...) . I don't like mixing key/value pair fields in the same block as arbitrary-lists-of-things, so the latter doesn't work for me, and the former adds more verbosity than I think is really needed. Second is how we handle declaring parallel groups in a stage, and related, how we handle existing parallel stages. I've played around with having parallel be able to contain either a list of stage or a list of group - or even a mix. Neither approach (exclusively stage or group , or non-exclusive) feels very smooth in implementation, though the syntax is pretty nice, especially for the mixed approach. I've also considered a new section, let's call it parallelGroups for now, where you could either have parallel or parallelGroups on a stage but not both. Much easier implementation-wise, better backwards compatibility (most notably for the editor, which wouldn't have to change how it handles parallel at all, just add parallelGroups support and make sure only one of the two can be used), worse syntax. So here are the options as I see it: Exclusive lists in parallel stage( 'just-stages' ) { parallel { // Just stages in this case stage( 'a' ) { ... } stage( 'b' ) { ... } ... } } stage( 'just-groups' ) { parallel { // Just groups in this case - note that we still need a group for just one stage group( 'first-group' ) { stage('a) { ... } } group( 'second-group' ) { stage( 'b' ) { ... } stage( 'c' ) { ... } } } Mixed lists in parallel stage( 'mixed-list' ) { parallel { stage( 'a' ) { ... } group( 'b-and-c' ) { stage( 'b' ) { ... } stage( 'c' ) { ... } } } } parallel and parallelGroups stage( 'parallel-single-stages' ) { parallel { stage( 'a' ) { ... } stage( 'b' ) { ... } } } stage( 'parallelGroups' ) { parallelGroups { group( 'first-group' ) { stage( 'a' ) { ... } } group( 'second-group' ) { stage( 'b' ) { ... } stage( 'c' ) { ... } } } } Thoughts? (ping kshultz michaelneale rsandell jamesdumay stephendonner rpocase hrmpw )

          Patrick Wolf added a comment -

          At first blush I don't think parallelGroups adds anything since group is required anyway.

          Is there any real difference between:

          stage('just-groups') {
            parallel {
             group('first-group') {
               stage('a) { ... }
             }
             group('second-group') {
               stage('b') { ... }
               stage('c') { ... }
            }
          }
          

          and 

          stage('parallelGroups') {
            parallelGroups {
              group('first-group') {
                stage('a') { ... }
              }
              group('second-group') {
                stage('b') { ... }
                stage('c') { ... }
              }
            }
          }
          

          I kind of like the former because we would be adding to existing parallel instead of adding a new higher level declaration. An alternative to group might be stageGroup to be a little more descriptive.

           

          Patrick Wolf added a comment - At first blush I don't think parallelGroups adds anything since group is required anyway. Is there any real difference between: stage( 'just-groups' ) { parallel { group( 'first-group' ) { stage('a) { ... } } group( 'second-group' ) { stage( 'b' ) { ... } stage( 'c' ) { ... } } } and  stage( 'parallelGroups' ) { parallelGroups { group( 'first-group' ) { stage( 'a' ) { ... } } group( 'second-group' ) { stage( 'b' ) { ... } stage( 'c' ) { ... } } } } I kind of like the former because we would be adding to existing parallel instead of adding a new higher level declaration. An alternative to group might be stageGroup to be a little more descriptive.  

            abayer Andrew Bayer
            banst Bastien Arata
            Votes:
            66 Vote for this issue
            Watchers:
            103 Start watching this issue

              Created:
              Updated:
              Resolved: