Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33185

Visualize parallel steps within a Pipeline Stage

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Support for visualizing parallel steps is unplanned

      A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

      The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

      There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.

      From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

      For the time being solving this problem has been delayed by CloudBees. However, we encourage any community members who are interested in solving this problem to pick it up.

      Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called Blue Ocean.

      Blue Ocean offers an entirely new way of visualizing Pipelines including parallel steps , and makes it much easier to navigate the Pipelines log by Stage and Step.

      We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

      You can find out more about Blue Ocean at https://jenkins.io/projects/blueocean/

      Thanks,
      James Dumay
      jdumay@cloudbees.com

      Original Request
      1) Parallel steps are the common workflow case
      2) Build logs for parallel steps are not informative
      3) A user would expect Stage View to somehow address the issue

      Current state:

      • stage declarations within parallel sections are not being considered as parallel
      • steps within one parallelized step are being mixed => no info in the view (see the screenshot)

      Test script (move stage declarations where you want):

      for (int i=0; i< 2; ++i) {
        stage "Stage #"+i
        print 'Hello, world $i!'
      }
      
      stage "Stage Parallel"
      def branches = [:]
      for (int i = 0; i < numHelloMessages.toInteger(); i++) {
        branches["split${i}"] = {
          stage "Stage parallel- #"+i
          node('remote') {
           echo  'Starting sleep'
           sleep 10
           echo  'Finished sleep'
          }
        }
      }
      parallel branches
      

      comment from JGlick:
      So concretely: the stage view would, for a given build (row), continue to display a linear sequence of stages (columns) with their associated times and so on; but each cell (stage of a build) would be some rendering of an st-planar graph, with structure determined according to the presence of steps of interest like parallel or a proposed label. Sort of like the existing Workflow Steps, but rendered as a graph rather than a tree table, restricted to displaying certain steps, and divided by stage.
      A simpler request would be to only display branches of one top-level parallel step, if any. (I.e., an st-planar graph which is either trivial—one vertex—or where all vertices other than the source and sink are directly connected to both the source and the sink.) This more limited UI would match that of the Build Pipeline plugin.
      Note that there is also a request JENKINS-29892 which would allow several stages to share concurrency semantics, but I think that need not have any effect on visualization; it would just be an internal change in the stage step.

        Attachments

        1. blue-ocean-parallel-generated-view.png
          blue-ocean-parallel-generated-view.png
          76 kB
        2. blue-ocean-parallel-WORKS.png
          blue-ocean-parallel-WORKS.png
          8 kB
        3. parallel.png
          parallel.png
          58 kB
        4. pipeline.png
          pipeline.png
          15 kB
        5. Screen Shot 2016-11-21 at 15.41.33.png
          Screen Shot 2016-11-21 at 15.41.33.png
          14 kB
        6. stage-duration.png
          stage-duration.png
          52 kB
        7. stages.png
          stages.png
          359 kB
        8. successful-pipeline.png
          successful-pipeline.png
          140 kB

          Issue Links

            Activity

            Hide
            jamesdumay James Dumay added a comment - - edited

            I am asking for feedback so we can move forward on this issue

            Tables make a lot of sense when you want scanability and comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that scanability and start making the view negatively information dense.

            We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons).

            For each run of the pipeline we would show something like:

            Example of stage duration comparison graph. X=Pipeline runs going forward in time and Y=duration of each stage

            Show
            jamesdumay James Dumay added a comment - - edited I am asking for feedback so we can move forward on this issue Tables make a lot of sense when you want scanability and comparison between each row (for example, comparing duration, which seems to be a key feature of the stage view). Adding parallel steps in a "cell" will destroy that scanability and start making the view negatively information dense. We've been thinking of representing single pipeline results like the screenshot below. Timings would either be a detail on hover or shown on a seperate graph (graphs are way better at visualising time comparisons). For each run of the pipeline we would show something like: Example of stage duration comparison graph. X=Pipeline runs going forward in time and Y=duration of each stage
            Hide
            rtyler R. Tyler Croy added a comment -

            James Dumay I like your graph/timeline proposal, it makes sense to me.

            Show
            rtyler R. Tyler Croy added a comment - James Dumay I like your graph/timeline proposal, it makes sense to me.
            Hide
            svanoort Sam Van Oort added a comment -

            I like the notion of something like the first one as an expansion option to get details. I think it might make sense to integrate it like so: hook into the front-end extension point API (JS), add a widget to indicate presence of parallel stages within a stage, then mouseover or click expands to the mini flow-graph for a stage.

            Show
            svanoort Sam Van Oort added a comment - I like the notion of something like the first one as an expansion option to get details. I think it might make sense to integrate it like so: hook into the front-end extension point API (JS), add a widget to indicate presence of parallel stages within a stage, then mouseover or click expands to the mini flow-graph for a stage.
            Hide
            jglick Jesse Glick added a comment -

            While such a view could be implemented using the currently available flow graphs, restricting itself to display of parallel branches, JENKINS-26107 would offer more flexibility.

            Show
            jglick Jesse Glick added a comment - While such a view could be implemented using the currently available flow graphs, restricting itself to display of parallel branches, JENKINS-26107 would offer more flexibility.
            Hide
            svanoort Sam Van Oort added a comment - - edited
            Show
            svanoort Sam Van Oort added a comment - - edited @ Jesse Glick
            Hide
            phiche Philip Cheong added a comment -

            I would be happy with something more simple such as this:

            https://wiki.jenkins-ci.org/download/attachments/69764097/screenshot2.png?version=1&modificationDate=1400172472000

            ...where the box indicating the stage simply displays the number of parallel steps that it will run. Bonus points for each step having it's own progress bar!

            Show
            phiche Philip Cheong added a comment - I would be happy with something more simple such as this: https://wiki.jenkins-ci.org/download/attachments/69764097/screenshot2.png?version=1&modificationDate=1400172472000 ...where the box indicating the stage simply displays the number of parallel steps that it will run. Bonus points for each step having it's own progress bar!
            Hide
            michaelneale Michael Neale added a comment -

            I am not sure Philip if those items inside the stages in that deploy pipeline view are parallel or just jobs (steps?)

            Show
            michaelneale Michael Neale added a comment - I am not sure Philip if those items inside the stages in that deploy pipeline view are parallel or just jobs (steps?)
            Hide
            mario_goorden Mario Goorden added a comment -

            It would be nice if each job has it's own 'live' logging view which is accessible from the single pipeline view as suggested by James Dumay.
            Now all logging of each parallel job end up in one log file, which makes analyzing a specific job tricky.

            Show
            mario_goorden Mario Goorden added a comment - It would be nice if each job has it's own 'live' logging view which is accessible from the single pipeline view as suggested by James Dumay. Now all logging of each parallel job end up in one log file, which makes analyzing a specific job tricky.
            Hide
            phiche Philip Cheong added a comment -

            Michael Neale isn't a "step" still just a step, regardless if it is defined to run in parallel or not?

            From a basic user-feedback perspective, what I want to see is:
            1. Tell me what stages are defined, which one is running now, and what time it has taken or estimated to complete. (I think the existing graphics and animations are all adequate in this area).
            2. If there are multiple steps inside the stage, tell me what step we are on, what time it has taken or estimated to complete. (This is what I am missing)

            Since pipeline stage-progress works from left to right, I'd like to see step-progress displayed from top to bottom for that specific pipeline animation.

            It doesn't concern me that the pipeline animation would then become be a dynamic height (as it is already dynamic length). Maybe it's a good option to make a check-box in the job configuration that I can click that shows or hides the steps inside the stages.

            I tend to work with large and complex pipelines (Currently on a project with > 880 git repos) so when I'm introducing CI/CD to different audiences these graphical front ends are really important when it comes to demo time so that the white-board concepts make sense with what I'm creating in the tools.

            Show
            phiche Philip Cheong added a comment - Michael Neale isn't a "step" still just a step, regardless if it is defined to run in parallel or not? From a basic user-feedback perspective, what I want to see is: 1. Tell me what stages are defined, which one is running now, and what time it has taken or estimated to complete. (I think the existing graphics and animations are all adequate in this area). 2. If there are multiple steps inside the stage, tell me what step we are on, what time it has taken or estimated to complete. (This is what I am missing) Since pipeline stage-progress works from left to right, I'd like to see step-progress displayed from top to bottom for that specific pipeline animation. It doesn't concern me that the pipeline animation would then become be a dynamic height (as it is already dynamic length). Maybe it's a good option to make a check-box in the job configuration that I can click that shows or hides the steps inside the stages. I tend to work with large and complex pipelines (Currently on a project with > 880 git repos) so when I'm introducing CI/CD to different audiences these graphical front ends are really important when it comes to demo time so that the white-board concepts make sense with what I'm creating in the tools.
            Hide
            uncodead Bruno Leitao added a comment -

            I missed this feature in my workflow. I have steps in parallel and i would like to see parallel in a same column

            Show
            uncodead Bruno Leitao added a comment - I missed this feature in my workflow. I have steps in parallel and i would like to see parallel in a same column
            Hide
            petergottinger Peter Gottinger added a comment -

            I would also love to see the visualization of parallel build steps in the plugin.

            Show
            petergottinger Peter Gottinger added a comment - I would also love to see the visualization of parallel build steps in the plugin.
            Hide
            jamesdumay James Dumay added a comment - - edited

            We will be releasing a new pipeline visualisation that handles parallel this month into open source. Its not related to the stage view plugin but we could look at improving it to fit here too. Ill update this ticket when we can show more.

            Show
            jamesdumay James Dumay added a comment - - edited We will be releasing a new pipeline visualisation that handles parallel this month into open source. Its not related to the stage view plugin but we could look at improving it to fit here too. Ill update this ticket when we can show more.
            Hide
            danagoyette Dana Goyette added a comment -

            I hope you'll also include visualization for steps within a parallel block.

            I'd like to be able to replace Matrix Jobs with equivalent workflows / pipelines, and that requires one big parallel block with a bunch of rows of the same steps (with different parameters).

            Show
            danagoyette Dana Goyette added a comment - I hope you'll also include visualization for steps within a parallel block. I'd like to be able to replace Matrix Jobs with equivalent workflows / pipelines, and that requires one big parallel block with a bunch of rows of the same steps (with different parameters).
            Hide
            abayer Andrew Bayer added a comment -

            Dana Goyette - that's exactly what this is about. =)

            Show
            abayer Andrew Bayer added a comment - Dana Goyette - that's exactly what this is about. =)
            Hide
            danagoyette Dana Goyette added a comment -

            Some of the concepts above (such as the "on click" screenshot) just didn't seem tailored well to that case; hence my comment. That's all.

            Show
            danagoyette Dana Goyette added a comment - Some of the concepts above (such as the "on click" screenshot) just didn't seem tailored well to that case; hence my comment. That's all.
            Hide
            johnwynne John Wynne added a comment - - edited

            I agree with Dana Goyette here. Not sure how feasible/scalable it would be, but here's a mockup that I think could work well with the Pipeline Stage View Plugin

            Show
            johnwynne John Wynne added a comment - - edited I agree with Dana Goyette here. Not sure how feasible/scalable it would be, but here's a mockup that I think could work well with the Pipeline Stage View Plugin
            Hide
            svanoort Sam Van Oort added a comment -

            John Wynne I really like the look of this – I was planning to go more or less that route when adding this (I am still working on some of the backend components needed to support this functionality at an API level).

            I might play with the formatting a bit to ensure the parallel branches within a stage are still associated to the stage they belong to (maybe create an outline around the stage).

            Show
            svanoort Sam Van Oort added a comment - John Wynne I really like the look of this – I was planning to go more or less that route when adding this (I am still working on some of the backend components needed to support this functionality at an API level). I might play with the formatting a bit to ensure the parallel branches within a stage are still associated to the stage they belong to (maybe create an outline around the stage).
            Hide
            Carbon Kieran Webber added a comment -

            John Wynne That's the exact image i had mentally when first looking around this subject so +1 for a design along the lines of that

            Show
            Carbon Kieran Webber added a comment - John Wynne That's the exact image i had mentally when first looking around this subject so +1 for a design along the lines of that
            Hide
            perilousapricot Andrew Melo added a comment -

            How would that visualization look if you have tons of things running in parallel that aren't necessarily related? For instance, I run the packaging step immediately, with the tests in parallel (to reduce the overall latency). Would I end up with one very tall column?

            Show
            perilousapricot Andrew Melo added a comment - How would that visualization look if you have tons of things running in parallel that aren't necessarily related? For instance, I run the packaging step immediately, with the tests in parallel (to reduce the overall latency). Would I end up with one very tall column?
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Melo At first it would probably do that for the initial version (tall windows), just to get a minimum-viable implementation out.
            One enhancement beyond that would be to switch to an accordion view if the number of parallel branches was bigger than some threshold, showing just a count of parallel branches at the top and a button to expand.

            Alternately, we might show basic stats or icons at the top of the rollup bar to give a fast visual indicator of branch status, something like: Succeeded / Running / Failed / Total - that approach scales well to numerous branches... but for now, high concurrency within parallel blocks is somewhat limited.

            Show
            svanoort Sam Van Oort added a comment - Andrew Melo At first it would probably do that for the initial version (tall windows), just to get a minimum-viable implementation out. One enhancement beyond that would be to switch to an accordion view if the number of parallel branches was bigger than some threshold, showing just a count of parallel branches at the top and a button to expand. Alternately, we might show basic stats or icons at the top of the rollup bar to give a fast visual indicator of branch status, something like: Succeeded / Running / Failed / Total - that approach scales well to numerous branches... but for now, high concurrency within parallel blocks is somewhat limited.
            Hide
            svanoort Sam Van Oort added a comment -

            Worth noting if you do display parallel branches in this fashion: it is critical to sort parallel forks by name (sorted alphabetically) since the actual number of branches and evaluation order can be completely random, so we must impose absolute ordering. This is because branches are received in a hashmap, and iterator order is dependent on the hash.

            Show
            svanoort Sam Van Oort added a comment - Worth noting if you do display parallel branches in this fashion: it is critical to sort parallel forks by name (sorted alphabetically) since the actual number of branches and evaluation order can be completely random, so we must impose absolute ordering. This is because branches are received in a hashmap, and iterator order is dependent on the hash.
            Hide
            barryoneill Barry O'Neill added a comment -

            Is there any possibility that this issue will be picked up? We're stuck in the old pipeline world and would love to use the new 2.0 pipeline, but people here are very fond of the visual representation of the parallel jobs that the older pipeline plugins offer.

            Show
            barryoneill Barry O'Neill added a comment - Is there any possibility that this issue will be picked up? We're stuck in the old pipeline world and would love to use the new 2.0 pipeline, but people here are very fond of the visual representation of the parallel jobs that the older pipeline plugins offer.
            Hide
            johnament John Ament added a comment -

            In addition, since the old pipeline doesn't work in Jenkins 2, that blocks upgrades to newer Jenkins releases

            Show
            johnament John Ament added a comment - In addition, since the old pipeline doesn't work in Jenkins 2, that blocks upgrades to newer Jenkins releases
            Hide
            gitt Slawa Giterman added a comment - - edited

            You can try the new Jenkins Pipeline GUI: https://jenkins.io/projects/blueocean/. It has already support for the parallel steps visualization.

            The Project Blue Ocean is still in alpha, but is under active development: https://jenkins.io/blog/2016/07/19/blue-ocean-update/. It is available in the experimental update center: http://updates.jenkins-ci.org/experimental/update-center.json.

            GitHub: https://github.com/jenkinsci/blueocean-plugin

            Show
            gitt Slawa Giterman added a comment - - edited You can try the new Jenkins Pipeline GUI: https://jenkins.io/projects/blueocean/ . It has already support for the parallel steps visualization. The Project Blue Ocean is still in alpha, but is under active development: https://jenkins.io/blog/2016/07/19/blue-ocean-update/ . It is available in the experimental update center: http://updates.jenkins-ci.org/experimental/update-center.json . GitHub: https://github.com/jenkinsci/blueocean-plugin
            Hide
            barryoneill Barry O'Neill added a comment -

            Wow, looks slick. I'm not sure it's stable enough for us to use for our main CI just yet, but that definitely is more along the lines of what we're looking for. Thanks for the update!

            Show
            barryoneill Barry O'Neill added a comment - Wow, looks slick. I'm not sure it's stable enough for us to use for our main CI just yet, but that definitely is more along the lines of what we're looking for. Thanks for the update!
            Hide
            svanoort Sam Van Oort added a comment - - edited

            John Ament Wait, what's the issue you're experiencing with Jenkins 2.0? This is supposed to be 100% compatible, and I haven't seen any JIRAs opened about an issue. If there is one, it will be a very high priority fix.

            Barry O'Neill I've updated status to reflect that we are indeed working towards this. It's not forgotten, not abandoned, just far more work to implement than anyone would expect without seeing the code or edge cases. I think I've currently invested ~1-2 months.

            The challenge is that we did not have a common set of APIs for decomposing the directed acyclic graphs from a pipeline execution, and have hit multiple problems like this one that are painful without them. So, I've had to write those components to provide the functionality we need. And they need to handle a lot of nasty, complex edge cases to work correctly – for example: nested parallel blocks, where both parallel blocks are incomplete.

            The current plan is that the BlueOcean UI linked above will also consume the same APIs to provide additional functionality.

            Show
            svanoort Sam Van Oort added a comment - - edited John Ament Wait, what's the issue you're experiencing with Jenkins 2.0? This is supposed to be 100% compatible, and I haven't seen any JIRAs opened about an issue. If there is one, it will be a very high priority fix. Barry O'Neill I've updated status to reflect that we are indeed working towards this. It's not forgotten, not abandoned, just far more work to implement than anyone would expect without seeing the code or edge cases. I think I've currently invested ~1-2 months. The challenge is that we did not have a common set of APIs for decomposing the directed acyclic graphs from a pipeline execution, and have hit multiple problems like this one that are painful without them. So, I've had to write those components to provide the functionality we need. And they need to handle a lot of nasty, complex edge cases to work correctly – for example: nested parallel blocks, where both parallel blocks are incomplete. The current plan is that the BlueOcean UI linked above will also consume the same APIs to provide additional functionality.
            Hide
            raphink Raphaël PINSON added a comment -

            Slawa Giterman the Jenkins Pipeline GUI is great, but it still doesn't show steps inside each parallel branch, even though the syntax is accepted. Do you know if that is planned?

            Show
            raphink Raphaël PINSON added a comment - Slawa Giterman the Jenkins Pipeline GUI is great, but it still doesn't show steps inside each parallel branch, even though the syntax is accepted. Do you know if that is planned?
            Hide
            clockman Piotr Zegar added a comment -

            I see 2 issues with that plugin in latest version:

            1. Step time is invalid, like it stop counting when all paraell tasks are started, it looks like step finish but in real job is still waiting for finish of parallel taks until move to next step.
            2. No option to scroll to see older builds, it shows only last ~20, but cannot see older one. Would be nice to see same info when entering a specific build page.

            Except that, I like it..

            Show
            clockman Piotr Zegar added a comment - I see 2 issues with that plugin in latest version: 1. Step time is invalid, like it stop counting when all paraell tasks are started, it looks like step finish but in real job is still waiting for finish of parallel taks until move to next step. 2. No option to scroll to see older builds, it shows only last ~20, but cannot see older one. Would be nice to see same info when entering a specific build page. Except that, I like it..
            Hide
            svanoort Sam Van Oort added a comment -

            Piotr Zegar Do you mean the Blue-Ocean plugin? Stage View? Pipeline view plugin? Build Pipeline Plugin?

            These are all different things.

            Show
            svanoort Sam Van Oort added a comment - Piotr Zegar Do you mean the Blue-Ocean plugin? Stage View? Pipeline view plugin? Build Pipeline Plugin? These are all different things.
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            What would be the current expect way to define a job with multiple parallel tasks if we cannot use stage inside it? How we are going to distinguish between "parallel-build-foo" and "parallel-build-bar"?

            Show
            ssbarnea Sorin Sbarnea added a comment - What would be the current expect way to define a job with multiple parallel tasks if we cannot use stage inside it? How we are going to distinguish between "parallel-build-foo" and "parallel-build-bar"?
            Hide
            aburdukovskiy Anton B added a comment -

            I imagine that since the parallel step takes a map, the keys can act as identifiers for the parallel tasks.

            Show
            aburdukovskiy Anton B added a comment - I imagine that since the parallel step takes a map, the keys can act as identifiers for the parallel tasks.
            Hide
            svanoort Sam Van Oort added a comment -

            Sorin Sbarnea The answer from Anton B is correct. The block-scoped version of the stage step, i.e.

            stage ('myStageName') {
              // do stuff here
            }
            

            can also be included in parallels. However there is not a UI visualization for that approach yet.

            Show
            svanoort Sam Van Oort added a comment - Sorin Sbarnea The answer from Anton B is correct. The block-scoped version of the stage step, i.e. stage ( 'myStageName' ) { // do stuff here } can also be included in parallels. However there is not a UI visualization for that approach yet.
            Hide
            michaelneale Michael Neale added a comment -

            The name of the parallel "branches" do indeed look like stages, and provide and identifier (this shows up int he log as a prefix, and also in blue ocean visualisation as branches of execution).

            Sam Van Oort I wouldn't recomment putting stages inside of things until we decide how we want to visualise it.

            Show
            michaelneale Michael Neale added a comment - The name of the parallel "branches" do indeed look like stages, and provide and identifier (this shows up int he log as a prefix, and also in blue ocean visualisation as branches of execution). Sam Van Oort I wouldn't recomment putting stages inside of things until we decide how we want to visualise it.
            Hide
            svanoort Sam Van Oort added a comment -

            Indeed – the branch name is the identifier in this case, it's up in the air if stages inside parallels will be supported in the UI or not. The syntax allows it, but just because you can doesn't mean you should!

            Show
            svanoort Sam Van Oort added a comment - Indeed – the branch name is the identifier in this case, it's up in the air if stages inside parallels will be supported in the UI or not. The syntax allows it, but just because you can doesn't mean you should!
            Hide
            rkuba Jakub Rusiecki added a comment - - edited

            To recap visualization needs I'd suggest swimlines (it is simpler that rectangles in rectangles..) :
            1. branches of a stage
            Stage can fork into parallel branches (as described above) - these need parallel visualization by a) names described in parallel statement and b) their times.
            Stage time shall be actual time incl. branches.
            Visualization should make swimline for each branch below stage they fork.
            2. branch stages
            Each branch may have stages - time of branch shall include sum of it's steps and stages.
            Visualization should make swimline for each stage below branch they are included in.
            3. sub-stages
            Each stage (e.g. A) can have sub-stages (e.g. A1 and A2) - Visualization of substages (A1, A2) as one swimline for all substages one-after-another below main stage (A) so to show that substage times are included in stage time.

            Show
            rkuba Jakub Rusiecki added a comment - - edited To recap visualization needs I'd suggest swimlines (it is simpler that rectangles in rectangles..) : 1. branches of a stage Stage can fork into parallel branches (as described above) - these need parallel visualization by a) names described in parallel statement and b) their times. Stage time shall be actual time incl. branches. Visualization should make swimline for each branch below stage they fork. 2. branch stages Each branch may have stages - time of branch shall include sum of it's steps and stages. Visualization should make swimline for each stage below branch they are included in. 3. sub-stages Each stage (e.g. A) can have sub-stages (e.g. A1 and A2) - Visualization of substages (A1, A2) as one swimline for all substages one-after-another below main stage (A) so to show that substage times are included in stage time.
            Hide
            lukasio10 Lukasz Gawel added a comment -

            Do we have a timeline for this fix?

            Show
            lukasio10 Lukasz Gawel added a comment - Do we have a timeline for this fix?
            Hide
            elgalu Leo Gallucci added a comment -

            This missing feature might be one of main reasons why Jenkins is on HOLD in one of the most popular tech radars:
            https://www.thoughtworks.com/radar/tools

            Show
            elgalu Leo Gallucci added a comment - This missing feature might be one of main reasons why Jenkins is on HOLD in one of the most popular tech radars: https://www.thoughtworks.com/radar/tools
            Hide
            jamesdumay James Dumay added a comment - - edited

            Leo Gallucci take a look at Blue Ocean - we're building an entirely new user experience for CD pipelines, including a visual editor for Pipeline.

            Here's a screenshot that includes parallel visualisation:

            Show
            jamesdumay James Dumay added a comment - - edited Leo Gallucci take a look at Blue Ocean - we're building an entirely new user experience for CD pipelines, including a visual editor for Pipeline . Here's a screenshot that includes parallel visualisation:
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            Leo Gallucci Please remember that ThoughtWorks has its own CI/CD solutions so I would count their tech radar a biased one. Still, I have to confess that I partially support your view, I would love to be able to migrate OpenStack build jobs to pipelines but I doubt it would be possible in the next 1-2 years, and that's because they are not mature enough and because there were some design decisions regarding not supporting old jobs, making progressive transitions hardly possible.

            Shortly, future is bright for small companies with simple workflows as they would be able to benefit from Jenkins pipelines. Others are kinda screwed as I don't see any migration scenario working at this moment.

            Show
            ssbarnea Sorin Sbarnea added a comment - Leo Gallucci Please remember that ThoughtWorks has its own CI/CD solutions so I would count their tech radar a biased one. Still, I have to confess that I partially support your view, I would love to be able to migrate OpenStack build jobs to pipelines but I doubt it would be possible in the next 1-2 years, and that's because they are not mature enough and because there were some design decisions regarding not supporting old jobs, making progressive transitions hardly possible. Shortly, future is bright for small companies with simple workflows as they would be able to benefit from Jenkins pipelines. Others are kinda screwed as I don't see any migration scenario working at this moment.
            Hide
            elgalu Leo Gallucci added a comment -

            Yes Sorin Sbarnea I also thought about them being biased, I assisted their tech radar presentation and the topic came, the presenter assured they are not biased and they make a big effort to be objective and explained all the reasons behind it.

            James Dumay Blue Ocean looked promising, I installed it. Why a visual editor? shouldn't the view be generated out of the Jenkinsfile? I thought we want pipeline as code.
            So I tried to write `parallel` Jenkinsfiles and the generated view looks weird, both examples:

            Left example

            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              parallel (
               "chrome" : { println 'running tests in chrome' },
               "firefox" : { println 'running tests in firefox' },
               "edge" : { println 'running tests in edge' }
              )
            }
            

            Right example

            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              parallel (
               "chrome" : { stage('chrome') { println 'running tests in chrome' } },
               "firefox" : { stage('firefox') { println 'running tests in firefox' } },
               "edge" : { stage('edge') { println 'running tests in edge' } }
              )
            }
            

            Show
            elgalu Leo Gallucci added a comment - Yes Sorin Sbarnea I also thought about them being biased, I assisted their tech radar presentation and the topic came, the presenter assured they are not biased and they make a big effort to be objective and explained all the reasons behind it. James Dumay Blue Ocean looked promising, I installed it. Why a visual editor? shouldn't the view be generated out of the Jenkinsfile? I thought we want pipeline as code. So I tried to write `parallel` Jenkinsfiles and the generated view looks weird, both examples: Left example node() { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } parallel ( "chrome" : { println 'running tests in chrome' }, "firefox" : { println 'running tests in firefox' }, "edge" : { println 'running tests in edge' } ) } Right example node() { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } parallel ( "chrome" : { stage( 'chrome' ) { println 'running tests in chrome' } }, "firefox" : { stage( 'firefox' ) { println 'running tests in firefox' } }, "edge" : { stage( 'edge' ) { println 'running tests in edge' } } ) }
            Hide
            crummy malcolm crum added a comment -

            Can you try this instead:

            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              stage('Test') {
               parallel (
               "chrome" : { println 'running tests in chrome' },
               "firefox" : { println 'running tests in firefox' },
               "edge" : { println 'running tests in edge' }
              )
              }
            }
            
            Show
            crummy malcolm crum added a comment - Can you try this instead: node() { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } stage( 'Test' ) { parallel ( "chrome" : { println 'running tests in chrome' }, "firefox" : { println 'running tests in firefox' }, "edge" : { println 'running tests in edge' } ) } }
            Hide
            elgalu Leo Gallucci added a comment -

            Yay! thanks malcolm crum (my bad)

            Show
            elgalu Leo Gallucci added a comment - Yay! thanks malcolm crum (my bad)
            Hide
            diemol Diego Molina added a comment -

            I guess it is better to read a bit how it works before Leo Gallucci

            Show
            diemol Diego Molina added a comment - I guess it is better to read a bit how it works before Leo Gallucci
            Hide
            battika Attila Strba added a comment -

            Guys one more question, as far as I understood from Jesse Glick post here https://issues.jenkins-ci.org/browse/JENKINS-28293?focusedCommentId=277255&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-277255 the parallel steps visualisation should be also supported if stages are nested under the parallel step since stage accepts block parameter, correct?

            Does this mean that the Blue Ocean plugin has to be still updated in order the visualisation displays it correctly?

            Show
            battika Attila Strba added a comment - Guys one more question, as far as I understood from Jesse Glick post here https://issues.jenkins-ci.org/browse/JENKINS-28293?focusedCommentId=277255&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-277255 the parallel steps visualisation should be also supported if stages are nested under the parallel step since stage accepts block parameter, correct? Does this mean that the Blue Ocean plugin has to be still updated in order the visualisation displays it correctly?
            Hide
            pjaytycy Pieter-Jan Busschaert added a comment -

            Does/will the visualization in BlueOcean support sub-stages in parallel?

            I think we have a very common use-case, but it's not really supported in the current BlueOcean gui:

            parallel (
             "win7-vs2012" : {node("vs2012") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} },
             "win10-vs2015" : {node ("vs2015") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} },
             "linux-gcc5" : {node ("gcc5") {stage("checkout") {...}; stage("build") {...}; stage("test") {...};} }
            )
            stage("email notifications") {...}
            
            Show
            pjaytycy Pieter-Jan Busschaert added a comment - Does/will the visualization in BlueOcean support sub-stages in parallel? I think we have a very common use-case, but it's not really supported in the current BlueOcean gui: parallel ( "win7-vs2012" : {node( "vs2012" ) {stage( "checkout" ) {...}; stage( "build" ) {...}; stage( "test" ) {...};} }, "win10-vs2015" : {node ( "vs2015" ) {stage( "checkout" ) {...}; stage( "build" ) {...}; stage( "test" ) {...};} }, "linux-gcc5" : {node ( "gcc5" ) {stage( "checkout" ) {...}; stage( "build" ) {...}; stage( "test" ) {...};} } ) stage( "email notifications" ) {...}
            Hide
            crummy malcolm crum added a comment - - edited

            Pieter-Jan Busschaert, I just tested the following code:

            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              stage('Test') {
               parallel (
             "win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
             "win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
             "linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
            )
              }
            }
            

            I didn't see the stages in the parallel steps. I don't think that's supported in blue ocean right now:

            Show
            crummy malcolm crum added a comment - - edited Pieter-Jan Busschaert , I just tested the following code: node() { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } stage( 'Test' ) { parallel ( "win7-vs2012" : { stage( "checkout" ) { }; stage( "build" ) { }; stage( "test" ) { } }, "win10-vs2015" : { stage( "checkout" ) { }; stage( "build" ) { }; stage( "test" ) { }}, "linux-gcc5" : { stage( "checkout" ) { }; stage( "build" ) { }; stage( "test" ) { } } ) } } I didn't see the stages in the parallel steps. I don't think that's supported in blue ocean right now:
            Hide
            jhack Giacomo Boccardo added a comment -
            Show
            jhack Giacomo Boccardo added a comment - According to the following tickets, parallel steps in Blue Ocean are not supported very well: https://issues.jenkins-ci.org/browse/JENKINS-39847 https://issues.jenkins-ci.org/browse/JENKINS-39464 https://issues.jenkins-ci.org/browse/JENKINS-39158
            Hide
            svanoort Sam Van Oort added a comment -

            Ping James Dumay and Vivek Pandey for questions/requests about Blue Ocean and parallels/nesting.

            Show
            svanoort Sam Van Oort added a comment - Ping James Dumay and Vivek Pandey for questions/requests about Blue Ocean and parallels/nesting.
            Hide
            jamesdumay James Dumay added a comment - - edited

            malcolm crum Pieter-Jan Busschaert nested stages (anywhere where there is a outer and inner stage) as you've suggested can't be visualised in Blue Ocean as we haven't been able to design a visualisation that meets all users requirements and expectations. There have been many solutions suggested but none that fit. You can follow and vote for it at JENKINS-38442. For the time being, we ignore the inner stages but show the steps within.

            Show
            jamesdumay James Dumay added a comment - - edited malcolm crum Pieter-Jan Busschaert nested stages (anywhere where there is a outer and inner stage) as you've suggested can't be visualised in Blue Ocean as we haven't been able to design a visualisation that meets all users requirements and expectations. There have been many solutions suggested but none that fit. You can follow and vote for it at JENKINS-38442 . For the time being, we ignore the inner stages but show the steps within.
            Hide
            jamesdumay James Dumay added a comment -

            Giacomo Boccardo the only ticket you have quoted that is unscheduled is JENKINS-39847. Aside, I think we do support parellel very well - Blue Ocean is the only place where parellel visualisation is supported so far

            As you can see, its a hard problem to solve visually when there are so many variations in the way that parallel can be used within Pipeline Script. The challenge with creating a visualization for Pipeline is to create a visualisation for turing complete programming language. That isn't practically possible under all circumstances.

            Show
            jamesdumay James Dumay added a comment - Giacomo Boccardo the only ticket you have quoted that is unscheduled is JENKINS-39847 . Aside, I think we do support parellel very well - Blue Ocean is the only place where parellel visualisation is supported so far As you can see, its a hard problem to solve visually when there are so many variations in the way that parallel can be used within Pipeline Script. The challenge with creating a visualization for Pipeline is to create a visualisation for turing complete programming language. That isn't practically possible under all circumstances.
            Hide
            michaelneale Michael Neale added a comment -

            malcolm crum Malcolm - could you sketch out what you would hope that the test stage above would look like? could be on a metaphorical back of a napkin (or literal napkin, although they can be hard to draw on).

            My guess is something like:

            Show
            michaelneale Michael Neale added a comment - malcolm crum Malcolm - could you sketch out what you would hope that the test stage above would look like? could be on a metaphorical back of a napkin (or literal napkin, although they can be hard to draw on). My guess is something like:
            Hide
            perilousapricot Andrew Melo added a comment -

            I understand completely it's hard to try and arrange a display for the results of a turing-complete programming language, but perhaps the compromise is allowing the programmer define where things should go, instead of having the language try to divine out the positioning post-facto. For instance (pseudo code, please be gentle)

            stage("checkout",0,0)

            { git checkout }

            stage("build",1,0)

            { build windows}

            stage("build",1,1)

            { build linux}

            stage("deploy",2,0)

            { deploy windows}

            stage("deploy",2,1)

            { deploy linux}

            Since the person writing the code knows (ostensibly) where things should go, you could just trust them to make the right grid-like display of each node/step

            Show
            perilousapricot Andrew Melo added a comment - I understand completely it's hard to try and arrange a display for the results of a turing-complete programming language, but perhaps the compromise is allowing the programmer define where things should go, instead of having the language try to divine out the positioning post-facto. For instance (pseudo code, please be gentle) stage("checkout",0,0) { git checkout } stage("build",1,0) { build windows} stage("build",1,1) { build linux} stage("deploy",2,0) { deploy windows} stage("deploy",2,1) { deploy linux} Since the person writing the code knows (ostensibly) where things should go, you could just trust them to make the right grid-like display of each node/step
            Hide
            jamesdumay James Dumay added a comment - - edited

            Andrew Melo being able to deliver a consistent visualization is one of the drivers of the new Declarative Pipeline syntax.

            Declarative Pipeline is structured. For example, you can only nest a single parellel block within a stage. There is no nesting of stages but parallel branches can have names that do what a lot of people on the comments of this ticket have done by defining a stage within a parellel.

            To use Leo Gallucci's example, in Pipeline Script:

            node() {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              parallel (
               "chrome" : { stage('chrome') { println 'running tests in chrome' } },
               "firefox" : { stage('firefox') { println 'running tests in firefox' } },
               "edge" : { stage('edge') { println 'running tests in edge' } }
              )
            }
            

            Would be the following in Declarative:

            pipeline {
              agent label:''" // same as node { ... }
              stages {
                stage('build') {
                  steps {
                    println 'I prepare the build for the parallel steps'
                  }
                }
                stage('browser testing') {
                  parallel (
                    'chrome': {
                      println 'running tests in chrome'
                    },
                    'firefox': {
                      println 'running tests in firefox'
                    },
                    'edge': {
                      println 'running tests in edge'
                    }
                  )
                }
              }
            }
            

            Now the more awesome part is that we can figure out how the Pipeline is displayed by reading the model vs executing it. That allows us to describe all the stages, steps and parallel branches before execution (making a linter feasible), build your own tooling to modify or construct Jenkinsfiles and even provides a foundation for the Jenkins project to build a Pipeline Editor.

            It may not be as flexible as the scripting variant that you've come to know but it will give much more consistent results across many of the pain points discussed here (however, you can use Script from a shared library in Declarative as an escape hatch).

            All this does not mean that Blue Ocean won't be trying to visualize your Jenkinsfiles written in Pipeline Script. It's important to us that it works for the majority case. However, we recognise that due to the nature of the beast there are going to be some things that just won't be possible to visualize when using Script.

            Show
            jamesdumay James Dumay added a comment - - edited Andrew Melo being able to deliver a consistent visualization is one of the drivers of the new Declarative Pipeline syntax. Declarative Pipeline is structured. For example, you can only nest a single parellel block within a stage . There is no nesting of stages but parallel branches can have names that do what a lot of people on the comments of this ticket have done by defining a stage within a parellel. To use Leo Gallucci 's example, in Pipeline Script: node() { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } parallel ( "chrome" : { stage( 'chrome' ) { println 'running tests in chrome' } }, "firefox" : { stage( 'firefox' ) { println 'running tests in firefox' } }, "edge" : { stage( 'edge' ) { println 'running tests in edge' } } ) } Would be the following in Declarative: pipeline { agent label:''" // same as node { ... } stages { stage( 'build' ) { steps { println 'I prepare the build for the parallel steps' } } stage( 'browser testing' ) { parallel ( 'chrome' : { println 'running tests in chrome' }, 'firefox' : { println 'running tests in firefox' }, 'edge' : { println 'running tests in edge' } ) } } } Now the more awesome part is that we can figure out how the Pipeline is displayed by reading the model vs executing it. That allows us to describe all the stages, steps and parallel branches before execution (making a linter feasible), build your own tooling to modify or construct Jenkinsfiles and even provides a foundation for the Jenkins project to build a Pipeline Editor . It may not be as flexible as the scripting variant that you've come to know but it will give much more consistent results across many of the pain points discussed here (however, you can use Script from a shared library in Declarative as an escape hatch). All this does not mean that Blue Ocean won't be trying to visualize your Jenkinsfiles written in Pipeline Script. It's important to us that it works for the majority case. However, we recognise that due to the nature of the beast there are going to be some things that just won't be possible to visualize when using Script.
            Hide
            perilousapricot Andrew Melo added a comment -

            I like it! My only hitch would be trying to describe the dependency that wasn't a series of stages (forgive my pseudo-code) where things would block unnecessarily:

            stage("checkout")

            { git checkout XXX }

            node("linux-ubuntu") { stage("compile")

            { XXX}

            }
            node("linux-centos") { stage ("compile")

            { XXX }

            ; stage ("test")

            { YYY }

            }

            It's a contrived example, but I currently do something like:
            state("checkout") {
            git checkout XXX
            }
            stage("compile") {
            parallel {
            node ("linux-ubuntu")

            { XXX }

            node ("linux-centos")

            { XXX }

            }
            stage ("test") {
            node ("linux-centos")

            { YYY }

            }

            which means every part of "compile" has to finish before the "test" phase happens, even if there are available executors. If there's a rewrite in a declarative language, I'd hope that the "test" parts could at least start before all the "compile" stages start. I know it's not what's exposed in the current or blue ocean implementations, but it would be nice to fire off additional nodes/stages without waiting on unnecessary parallel branches.

            Show
            perilousapricot Andrew Melo added a comment - I like it! My only hitch would be trying to describe the dependency that wasn't a series of stages (forgive my pseudo-code) where things would block unnecessarily: stage("checkout") { git checkout XXX } node("linux-ubuntu") { stage("compile") { XXX} } node("linux-centos") { stage ("compile") { XXX } ; stage ("test") { YYY } } It's a contrived example, but I currently do something like: state("checkout") { git checkout XXX } stage("compile") { parallel { node ("linux-ubuntu") { XXX } node ("linux-centos") { XXX } } stage ("test") { node ("linux-centos") { YYY } } which means every part of "compile" has to finish before the "test" phase happens, even if there are available executors. If there's a rewrite in a declarative language, I'd hope that the "test" parts could at least start before all the "compile" stages start. I know it's not what's exposed in the current or blue ocean implementations, but it would be nice to fire off additional nodes/stages without waiting on unnecessary parallel branches.
            Hide
            perilousapricot Andrew Melo added a comment -

            I think the visualization is a second-order optimization, but at least firing off the subordinate nodes would speed things up a lot. And ,as a script writer, if I could hint to you who goes where visually, that would simplify your job

            Show
            perilousapricot Andrew Melo added a comment - I think the visualization is a second-order optimization, but at least firing off the subordinate nodes would speed things up a lot. And ,as a script writer, if I could hint to you who goes where visually, that would simplify your job
            Hide
            jamesdumay James Dumay added a comment -

            Andrew Melo BTW you can wrap code in JIRA in {code} some code {code} to preserve formatting

            Show
            jamesdumay James Dumay added a comment - Andrew Melo BTW you can wrap code in JIRA in {code} some code {code} to preserve formatting
            Hide
            perilousapricot Andrew Melo added a comment -

            Sorry, James Durmay – I'm not super familiar with JIRA quoting. I can rewrite/edit if it makes it clearer.

            Show
            perilousapricot Andrew Melo added a comment - Sorry, James Durmay – I'm not super familiar with JIRA quoting. I can rewrite/edit if it makes it clearer.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Andrew Melo I'm glad you like it. We've got a more formal beta coming towards the end of the year for Declarative but if you want to try it out now please do!

            If you've got any specific use cases in mind, like the one above, you can file an issue on JIRA against the pipeline-model-definition component and Patrick Wolf will be in touch

            Show
            jamesdumay James Dumay added a comment - - edited Andrew Melo I'm glad you like it. We've got a more formal beta coming towards the end of the year for Declarative but if you want to try it out now please do! If you've got any specific use cases in mind, like the one above, you can file an issue on JIRA against the pipeline-model-definition component and Patrick Wolf will be in touch
            Hide
            perilousapricot Andrew Melo added a comment -

            Yessir, Java's not one of my proficiencies, and I failed at even trying to build Jenkins on my local (OSX) machine, but I'm willing to try again to help scratch my itch. I'll write a JIRA against the above component and give it another shot to try and help. Cheers!

            Show
            perilousapricot Andrew Melo added a comment - Yessir, Java's not one of my proficiencies, and I failed at even trying to build Jenkins on my local (OSX) machine, but I'm willing to try again to help scratch my itch. I'll write a JIRA against the above component and give it another shot to try and help. Cheers!
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            Can someone point me to some sample Jenkinsfile that runs two scripts in parallel while having this visible in the UI? I find extremely confusing to see lots and lots of nice screenshots of parallel pipelines but not even one code example for achieving it.

            Show
            ssbarnea Sorin Sbarnea added a comment - Can someone point me to some sample Jenkinsfile that runs two scripts in parallel while having this visible in the UI? I find extremely confusing to see lots and lots of nice screenshots of parallel pipelines but not even one code example for achieving it.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Sorin Sbarnea I think this might be what you are looking for:

            node {
              stage('Build') {
                println 'I prepare the build for the parallel steps'
              }
            
              stage('Browser Tests') {
                parallel (
                   "chrome" : { println 'running tests in chrome' },
                   "firefox" : { println 'running tests in firefox' },
                   "edge" : { println 'running tests in edge' }
                 )
              }
            }
            
            Show
            jamesdumay James Dumay added a comment - - edited Sorin Sbarnea I think this might be what you are looking for: node { stage( 'Build' ) { println 'I prepare the build for the parallel steps' } stage( 'Browser Tests' ) { parallel ( "chrome" : { println 'running tests in chrome' }, "firefox" : { println 'running tests in firefox' }, "edge" : { println 'running tests in edge' } ) } }
            Hide
            tknerr Torben Knerr added a comment - - edited

            Would love to see something like John Wynne proposed here as an intermediate solution for the current pipeline stage view:
            https://issues.jenkins-ci.org/browse/JENKINS-33185?focusedCommentId=258129&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-258129

            Blue Ocean is a huge update (30 plugins?) and probably not everyone can (or wants) to adopt it and has to stay with the classical jenkins UI for quite a while.

            Is there any ongoing work in PRs to implement John Wynne's proposal? Would it even have a chance to be accepted as an intermediate solution?

            Show
            tknerr Torben Knerr added a comment - - edited Would love to see something like John Wynne proposed here as an intermediate solution for the current pipeline stage view: https://issues.jenkins-ci.org/browse/JENKINS-33185?focusedCommentId=258129&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-258129 Blue Ocean is a huge update (30 plugins?) and probably not everyone can (or wants) to adopt it and has to stay with the classical jenkins UI for quite a while. Is there any ongoing work in PRs to implement John Wynne 's proposal? Would it even have a chance to be accepted as an intermediate solution?
            Hide
            pvohmann Peter Vohmann added a comment -

            We have a use case to drive parallel builds over multiple stages, since we code C++ for multiple architectures and environments.
            Would this be possible with the Pipeline Stage, or even Pipeline at its core?

            In the example you see

            • Multiple design targets of the code lead to parallel streams of execution,
            • The same stages occur in every stream.

            I expect to

            • Name a Stage not only on top level, but also within a node step.
            • Be able to *reuse* the Stage by name -> collect certain activity in the same Stage column.
            • visualize multiple steps under the same Stage title.
            • Within a Node(eg "B") step, switch stage -> split output from same node to two Stages.

            The Build and Test Stages would be active at the same time, depending on design stream.

            Output from this pipeline should be separated into exactly three Stages and activity within each stage.

            • Setup.Default.console
            • Build.web.console
            • Build.embedded.console
            • Test.web.console
            • Test.embedded.test-lowend.console
            • Test.embedded.test-highend.console
            node('A') {
                stage('Setup') {
                    println 'Preparing multiple builds'
                }
            }
            parallel (
                "web" : {
                    node('B') {
                        stage('Build') {
                            println 'Build web application'
                        }
                        stage('Test') {
                            println 'Selenium Tests'
                        }
                    }
                },
                "embedded" : {
                    node('C') {
                        stage('Build') {
                            println 'Build embedded solution'
                        }
                    }
                    stage('Test') {
                        parallel (
                            "test-lowend" : {
                                node('D1') {
                                    println 'Test lowend embedded'
                                }
                            }, 
                            "test-highend" : {
                                node('D2') {
                                    println 'Build highend embedded'
                                }
                            }
                        )
                    }
                }
            )
            
            Show
            pvohmann Peter Vohmann added a comment - We have a use case to drive parallel builds over multiple stages, since we code C++ for multiple architectures and environments. Would this be possible with the Pipeline Stage, or even Pipeline at its core? In the example you see Multiple design targets of the code lead to parallel streams of execution, The same stages occur in every stream. I expect to Name a Stage not only on top level, but also within a node step. Be able to * reuse * the Stage by name -> collect certain activity in the same Stage column. visualize multiple steps under the same Stage title. Within a Node(eg "B") step, switch stage -> split output from same node to two Stages. The Build and Test Stages would be active at the same time, depending on design stream. Output from this pipeline should be separated into exactly three Stages and activity within each stage. Setup.Default.console Build.web.console Build.embedded.console Test.web.console Test.embedded.test-lowend.console Test.embedded.test-highend.console node( 'A' ) { stage( 'Setup' ) { println 'Preparing multiple builds' } } parallel ( "web" : { node( 'B' ) { stage( 'Build' ) { println 'Build web application' } stage( 'Test' ) { println 'Selenium Tests' } } }, "embedded" : { node( 'C' ) { stage( 'Build' ) { println 'Build embedded solution' } } stage( 'Test' ) { parallel ( "test-lowend" : { node( 'D1' ) { println 'Test lowend embedded' } }, "test-highend" : { node( 'D2' ) { println 'Build highend embedded' } } ) } } )
            Hide
            jglick Jesse Glick added a comment -

            Is there any ongoing work in PRs to implement John Wynne's proposal?

            Not that I am aware of.

            Would it even have a chance to be accepted as an intermediate solution?

            If you would like to take over ownership of this plugin.

            Show
            jglick Jesse Glick added a comment - Is there any ongoing work in PRs to implement John Wynne's proposal? Not that I am aware of. Would it even have a chance to be accepted as an intermediate solution? If you would like to take over ownership of this plugin.
            Hide
            svanoort Sam Van Oort added a comment -

            Put a bit less bluntly, what I believe Jesse Glick is trying to say is that (as per James Dumay's earlier comment) this one will probably need a community PR to implement it. Right now CloudBees is putting most of our front-end development effort behind Blue Ocean since it promises a better & easier path to this functionality.

            Stage view is maintained but right now bandwidth is limited due to other projects. Doing this properly would take more time than I have available to do it as a personal side project – and I'm not going to knock together a poor quality implementation and bring the overall plugin quality down.

            Show
            svanoort Sam Van Oort added a comment - Put a bit less bluntly, what I believe Jesse Glick is trying to say is that (as per James Dumay's earlier comment) this one will probably need a community PR to implement it. Right now CloudBees is putting most of our front-end development effort behind Blue Ocean since it promises a better & easier path to this functionality. Stage view is maintained but right now bandwidth is limited due to other projects. Doing this properly would take more time than I have available to do it as a personal side project – and I'm not going to knock together a poor quality implementation and bring the overall plugin quality down.
            Hide
            tknerr Torben Knerr added a comment -

            Sam Van Oort Jesse Glick thanks for the feedback!

            Good to hear that stage view is still maintained. Understood it needs a community PR. Would love to see it, but currently too short on time too :-/

            Show
            tknerr Torben Knerr added a comment - Sam Van Oort Jesse Glick thanks for the feedback! Good to hear that stage view is still maintained. Understood it needs a community PR. Would love to see it, but currently too short on time too :-/
            Hide
            trejkaz trejkaz added a comment - - edited

            Is Blue Ocean going to be in a usable state any time soon? At the moment, it just shows a progress bar which never finishes, so it's not really a viable workaround, let alone a good one. (On investigating this, it turns out a ticket was raised, but then resolved as "Duplicate" without actually fixing anything. Excellent!)

            As far as visualising the parallel stages, couldn't it be displayed serially like the existing serial stages already are? That would satisfy that desire to keep the table view.

            Although, I am a little in doubt over any supposed "benefits" to the table view. All I see is a gigantic table which forces me to scroll horizontally to see all the information. Even if I do scroll right to try and see more information, now the row headers roll off the left, so I have no idea which builds are red and which builds are green anyway. Showing it vertically might improve things, but I'm not sure it's a good visualisation in the first place, so I'm a little baffled as to why anyone would want to keep it. (I think perhaps as someone who actually has to use the software, I might have a different view of its prettiness to someone involved with developing it?)

            Show
            trejkaz trejkaz added a comment - - edited Is Blue Ocean going to be in a usable state any time soon? At the moment, it just shows a progress bar which never finishes, so it's not really a viable workaround, let alone a good one. (On investigating this, it turns out a ticket was raised, but then resolved as "Duplicate" without actually fixing anything. Excellent!) As far as visualising the parallel stages, couldn't it be displayed serially like the existing serial stages already are? That would satisfy that desire to keep the table view. Although, I am a little in doubt over any supposed "benefits" to the table view. All I see is a gigantic table which forces me to scroll horizontally to see all the information. Even if I do scroll right to try and see more information, now the row headers roll off the left, so I have no idea which builds are red and which builds are green anyway. Showing it vertically might improve things, but I'm not sure it's a good visualisation in the first place, so I'm a little baffled as to why anyone would want to keep it. (I think perhaps as someone who actually has to use the software, I might have a different view of its prettiness to someone involved with developing it?)
            Hide
            michaelneale Michael Neale added a comment -

            trejkaz "progress bar which never finishes" means the app installation is broken (would be errors in the browser console) - can you link the ticket to it? 

            Show
            michaelneale Michael Neale added a comment - trejkaz "progress bar which never finishes" means the app installation is broken (would be errors in the browser console) - can you link the ticket to it? 
            Hide
            trejkaz trejkaz added a comment - - edited

            I didn't think to check the browser console, since the page itself didn't indicate any kind of error. (Is there a ticket for that already?)

            Error: Must call ExtensionStore.init(\{ extensionData: array, typeInfoProvider: (type, cb) => ... }) first
            at ExtensionStore._initExtensionPointList (blueocean.js:49604)
            at ExtensionStore.init (blueocean.js:49449)
            at Object.init (blueocean.js:50102)
            at Object.execute (blueocean.js:143579)
            at APromise.<anonymous> (blueocean.js:144211)
            at Object.exports.make (blueocean.js:51393)
            at ___$$$___exec (blueocean.js:144208)
            at ___$$$___doBundleInit (blueocean.js:144296)
            at Object.1720.../../src/main/js/init (blueocean.js:144311)
            at s (blueocean.js:1)
            

            Searching for that error, the only occurrence I see in the tracker is JENKINS-44752 but it says that was "Fixed" too.

             

            Show
            trejkaz trejkaz added a comment - - edited I didn't think to check the browser console, since the page itself didn't indicate any kind of error. (Is there a ticket for that already?) Error: Must call ExtensionStore.init(\{ extensionData: array, typeInfoProvider: (type, cb) => ... }) first at ExtensionStore._initExtensionPointList (blueocean.js:49604) at ExtensionStore.init (blueocean.js:49449) at Object.init (blueocean.js:50102) at Object.execute (blueocean.js:143579) at APromise.<anonymous> (blueocean.js:144211) at Object.exports.make (blueocean.js:51393) at ___$$$___exec (blueocean.js:144208) at ___$$$___doBundleInit (blueocean.js:144296) at Object.1720.../../src/main/js/init (blueocean.js:144311) at s (blueocean.js:1) Searching for that error, the only occurrence I see in the tracker is JENKINS-44752 but it says that was "Fixed" too.  
            Hide
            jamesdumay James Dumay added a comment -

            trejkaz can you please open a new ticket for that error and attach a HAR file and support bundle?

            Show
            jamesdumay James Dumay added a comment - trejkaz can you please open a new ticket for that error and attach a HAR file and support bundle ?
            Hide
            michaelneale Michael Neale added a comment -

            usually that means some plugins have been removed/need to be updated (all blue ocean plugins).

            Show
            michaelneale Michael Neale added a comment - usually that means some plugins have been removed/need to be updated (all blue ocean plugins).
            Hide
            trejkaz trejkaz added a comment -

            It turned out that "Blue Ocean" plugin itself was never installed, but all the other parts somehow ended up being on the system even though nobody had manually installed them. I'm assuming Jenkins itself pulled in the individual parts without pulling in the parent. After manually adding it, the page finally loads, after never working the entire time since the link appeared on the site.

             

            Show
            trejkaz trejkaz added a comment - It turned out that "Blue Ocean" plugin itself was never installed, but all the other parts somehow ended up being on the system even though nobody had manually installed them. I'm assuming Jenkins itself pulled in the individual parts without pulling in the parent. After manually adding it, the page finally loads, after never working the entire time since the link appeared on the site.  
            Hide
            michaelneale Michael Neale added a comment -

            trejkaz ok that is good to hear - although I don't think other plugins yet depend on blue ocean parts, so not quite sure how it ended up doing that. There are other tickets on better generic error reporting that may make this error more obvious (although like with all error cases, edge cases mean it sometimes just wont work,but hopefully will at least give a hint).

            Show
            michaelneale Michael Neale added a comment - trejkaz ok that is good to hear - although I don't think other plugins yet depend on blue ocean parts, so not quite sure how it ended up doing that. There are other tickets on better generic error reporting that may make this error more obvious (although like with all error cases, edge cases mean it sometimes just wont work,but hopefully will at least give a hint).
            Hide
            trejkaz trejkaz added a comment -

            I'm not sure this view is really much better. I have to move down multiple pages of "pipelines" which aren't even pipelines to get to the only actual pipeline we have configured so far ... then when we do get to the one we want and look at some of the builds, the visualisation on each page doesn't even seem to show all the stages, just the one it's currently working on and then "End".

            Oh well.

            Show
            trejkaz trejkaz added a comment - I'm not sure this view is really much better. I have to move down multiple pages of "pipelines" which aren't even pipelines to get to the only actual pipeline we have configured so far ... then when we do get to the one we want and look at some of the builds, the visualisation on each page doesn't even seem to show all the stages, just the one it's currently working on and then "End". Oh well.
            Hide
            jeinstei Joshua Einstein-Curtis added a comment -

            I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.

             

             

            def jobMap (
                  some['thing'] = {node (label) {
                     stage1
                     stage2
                  } }
            )
             
            node {
               def jobs = jobMap()
               stage1
               stage2 {
                  parallel jobs
               }
            }
            

             

            Show
            jeinstei Joshua Einstein-Curtis added a comment - I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.     def jobMap (       some[ 'thing' ] = {node (label) { stage1 stage2       } } )   node { def jobs = jobMap()    stage1    stage2 {       parallel jobs    } }  
            Hide
            jk Jan Klass added a comment -

            Blue Ocean is unusable to me for its page load time alone.

            I can see how the tabular view can be useful to people.

            I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information.

            As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so.

            Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview.

            If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.

            Show
            jk Jan Klass added a comment - Blue Ocean is unusable to me for its page load time alone. I can see how the tabular view can be useful to people. I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information. As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so. Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview. If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.
            Hide
            jk Jan Klass added a comment - - edited

            With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument.

            I just saw the table disappear on me (presumably) because of this.

            /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??

            Show
            jk Jan Klass added a comment - - edited With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument. I just saw the table disappear on me (presumably) because of this. /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I would like to add this item into the Jenkins roadmap: https://jenkins.io/project/roadmap/

            Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

             

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - I would like to add this item into the Jenkins roadmap:  https://jenkins.io/project/roadmap/ Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.      
            Hide
            jglick Jesse Glick added a comment -

            Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI

            Makes no sense. B.O. already supports this.

            I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

            Also makes no sense. The limitation is specifically in the stage view plugin.

            Show
            jglick Jesse Glick added a comment - Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI Makes no sense. B.O. already supports this. I would also remove the component, because it is likely to become a separate plugin if it gets implemented. Also makes no sense. The limitation is specifically in the stage view plugin.

              People

              Assignee:
              cloudbees CloudBees Inc.
              Reporter:
              hrmpw Patrick Wolf
              Votes:
              210 Vote for this issue
              Watchers:
              215 Start watching this issue

                Dates

                Created:
                Updated: