Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39932

Support more arbitrary Pipeline definitions



    • Similar Issues:


      I think the idea of a declarative (and not imperative) pipeline is a lot easier on the writers end (and I assume the implementers end as well), but I think the implementation of several serial stages each fanning out to a number of parallel stages is limiting to the overall description of the dependencies between different steps/nodes. In my case (simplifying, please excuse pseudo-code), I'd like to do something like this:

      stage("checkout") { node () { git XXX } }
      stage("build") {
          parallel ("ubuntu") { node("ubuntu") { make XXX }},
                       ("centos") { node("centos") { make YYY }},
                       // tons more distributions here
      stage ("test") {
          node ("ubuntu") { test-ubuntu ZZZ }
      stage ("publish") { node() { publish.sh }

      Since I know the "test" stage only depends on the "ubuntu" part of the build stage, I'd like the test stage to fire without waiting on the whole "build" stage completes. From what I see, the parallel block waits for everything inside to complete, but I know the dependencies and can speed it up (in my case, there are a lot more nodes who match the testing than who don't). If I were to write it in bash, I would do something like:

      git checkout XXX
      build ubuntu &
      build centos &
      wait $TEST_PID
      wait # wait for all testing and bulding to complete

      This is a rough way to explain the dependency graph to bash by saying "wait on 'build ubuntu' before starting the testing, but centos can take as long as it wants" and "before going to the publish step, make sure everything else is done". In my case, all the build steps for different distributions take forever because there's a finite number of VMs that can be spun up, but there's an abundance of "ubuntu" nodes that can be used for testing and compiling. It slows things down a ton to have to wait for the entire parallel node to finish before moving on, when there are free executors in latter steps that can start running.


          Issue Links


            perilousapricot Andrew Melo created issue -
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Issue Type Bug [ 1 ] New Feature [ 2 ]
            abayer Andrew Bayer made changes -
            Link This issue relates to JENKINS-46809 [ JENKINS-46809 ]
            abayer Andrew Bayer made changes -
            Resolution Won't Fix [ 2 ]
            Status Open [ 1 ] Resolved [ 5 ]
            bitwiseman Liam Newman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]


              abayer Andrew Bayer
              perilousapricot Andrew Melo
              1 Vote for this issue
              4 Start watching this issue