Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39932

Support more arbitrary Pipeline definitions

XMLWordPrintable

      I think the idea of a declarative (and not imperative) pipeline is a lot easier on the writers end (and I assume the implementers end as well), but I think the implementation of several serial stages each fanning out to a number of parallel stages is limiting to the overall description of the dependencies between different steps/nodes. In my case (simplifying, please excuse pseudo-code), I'd like to do something like this:

      stage("checkout") { node () { git XXX } }
      stage("build") {
          parallel ("ubuntu") { node("ubuntu") { make XXX }},
                       ("centos") { node("centos") { make YYY }},
                       // tons more distributions here
      }
      stage ("test") {
          node ("ubuntu") { test-ubuntu ZZZ }
      }
      
      stage ("publish") { node() { publish.sh }
      

      Since I know the "test" stage only depends on the "ubuntu" part of the build stage, I'd like the test stage to fire without waiting on the whole "build" stage completes. From what I see, the parallel block waits for everything inside to complete, but I know the dependencies and can speed it up (in my case, there are a lot more nodes who match the testing than who don't). If I were to write it in bash, I would do something like:

      git checkout XXX
      build ubuntu &
      TEST_PID=$!
      build centos &
      wait $TEST_PID
      test-ubuntu
      wait # wait for all testing and bulding to complete
      publish
      

      This is a rough way to explain the dependency graph to bash by saying "wait on 'build ubuntu' before starting the testing, but centos can take as long as it wants" and "before going to the publish step, make sure everything else is done". In my case, all the build steps for different distributions take forever because there's a finite number of VMs that can be spun up, but there's an abundance of "ubuntu" nodes that can be used for testing and compiling. It slows things down a ton to have to wait for the entire parallel node to finish before moving on, when there are free executors in latter steps that can start running.

            abayer Andrew Bayer
            perilousapricot Andrew Melo
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: