[JENKINS-38442] View sequential stages in the pipeline visualization graph - Jenkins Jira

mohamed badran created issue - 2016-09-22 11:03

Michael Neale added a comment - 2016-09-22 22:14

Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph).

For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?

Michael Neale added a comment - 2016-09-22 22:14 Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph). For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?

James Dumay made changes - 2016-09-23 03:31

Field	Original Value	New Value
Issue Type	Bug [ 1 ]	Story [ 10002 ]

James Dumay made changes - 2016-09-23 03:31

Summary

Nested labelled blocks breaks blueocean

Support nested stages in the Pipeline Graph

James Dumay made changes - 2016-09-23 03:33

Description

I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:34

Description

{panel title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:45

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:46

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:46

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay added a comment - 2016-09-23 06:24

mbadran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?

James Dumay added a comment - 2016-09-23 06:24 mbadran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?

mohamed badran added a comment - 2016-09-25 09:06

Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script.

Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.

mohamed badran added a comment - 2016-09-25 09:06 Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script. Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.

Michael Neale added a comment - 2016-09-25 22:53

mbadran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?)

Perhaps we need to draw out what the options could be for this.

But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?

Michael Neale added a comment - 2016-09-25 22:53 mbadran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?) Perhaps we need to draw out what the options could be for this. But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?

James Dumay made changes - 2016-09-26 06:21

Epic Link

JENKINS-35750 [ 171713 ]

mohamed badran added a comment - 2016-09-26 08:53

michaelneale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing.

In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.

mohamed badran added a comment - 2016-09-26 08:53 michaelneale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing. In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.

Michael Neale added a comment - 2016-09-27 05:44 - edited

mbadran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit.

But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see...

jamesdumay perhaps this is a case for labelled steps of some kind?

Michael Neale added a comment - 2016-09-27 05:44 - edited mbadran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit. But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see... jamesdumay perhaps this is a case for labelled steps of some kind?

mohamed badran made changes - 2016-09-27 09:21

Attachment

Capture.JPG [ 34084 ]

mohamed badran added a comment - 2016-09-27 09:47

michaelneale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage.

For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages).

Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this

steps a/ab
shell script

steps a/xyz
shell script

These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like
shell script : (This is platform 1)
shell script: (This is platform 2)

So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.

mohamed badran added a comment - 2016-09-27 09:47 michaelneale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage. For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages). Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this steps a/ab shell script steps a/xyz shell script These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like shell script : (This is platform 1) shell script: (This is platform 2) So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.

James Dumay made changes - 2016-09-28 05:04

Attachment

blueocean.sketch 2016-09-28 15-03-57.png [ 34100 ]

James Dumay added a comment - 2016-09-28 05:06

One way we could make this work is:

Where we have a Pipeline like

  stage ("Build") {
    stage ("Inner 1") {
       ....
    }
    stage ("Inner 2") {
        ....
    }
  }

Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.

James Dumay added a comment - 2016-09-28 05:06 One way we could make this work is: Where we have a Pipeline like stage ( "Build" ) { stage ( "Inner 1" ) { .... } stage ( "Inner 2" ) { .... } } Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.

mohamed badran added a comment - 2016-09-28 09:11

jamesdumay and michaelneale

Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.

mohamed badran added a comment - 2016-09-28 09:11 jamesdumay and michaelneale Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.

James Dumay added a comment - 2016-09-29 00:06

Im still hesitant because we have no idea what pipelines like this should look like

stage ("outer") {
    stage ("inner") {
       stage ("most inner") {
          stage ("turtles all the way down") {
             ...
          }
       }
    }
  }

James Dumay added a comment - 2016-09-29 00:06 Im still hesitant because we have no idea what pipelines like this should look like stage ( " outer " ) { stage ( " inner " ) { stage ( "most inner " ) { stage ( "turtles all the way down" ) { ... } } } }

Michael Neale added a comment - 2016-09-29 00:28 - edited

Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config.

I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.

Michael Neale added a comment - 2016-09-29 00:28 - edited Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config. I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.

mohamed badran added a comment - 2016-09-29 09:11

michaelneale

Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything.

jamesdumay

Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like:

Steps Outer
Inner Stage inner
Inner Stage inner/most inner

I believe by this way users will always be able to see things clearly

mohamed badran added a comment - 2016-09-29 09:11 michaelneale Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything. jamesdumay Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like: Steps Outer Inner Stage inner Inner Stage inner/most inner I believe by this way users will always be able to see things clearly

Kaden Wilkinson made changes - 2016-10-03 22:14

Attachment

Screen Shot 2016-10-03 at 4.13.46 PM.png [ 34227 ]

Kaden Wilkinson added a comment - 2016-10-03 22:15

I think the reason many are wanting to even have nested stages is to help solve this problem:

We would like to give these shell scripts a more descriptive name but are unable to.

Kaden Wilkinson added a comment - 2016-10-03 22:15 I think the reason many are wanting to even have nested stages is to help solve this problem: We would like to give these shell scripts a more descriptive name but are unable to.

Michael Neale added a comment - 2016-10-03 23:11 - edited

kdawgwilk exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?

Michael Neale added a comment - 2016-10-03 23:11 - edited kdawgwilk exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?

Kaden Wilkinson added a comment - 2016-10-03 23:25 - edited

I would like to have a syntax in the Jenkinfile to be able to say:

sh([cmd: 'script.sh', name: 'Deploy script'])

or maybe something more generic that would work for all build steps?

step('Deploy script') {
    sh 'script.sh'
}

Kaden Wilkinson added a comment - 2016-10-03 23:25 - edited I would like to have a syntax in the Jenkinfile to be able to say: sh([cmd: 'script.sh' , name: 'Deploy script' ]) or maybe something more generic that would work for all build steps? step( 'Deploy script' ) { sh 'script.sh' }

Kaden Wilkinson added a comment - 2016-10-03 23:28

Even the name of the script being called would help eg.

sh 'script.sh'

if the step was displayed as "script.sh" it would be much better than it is right now

Kaden Wilkinson added a comment - 2016-10-03 23:28 Even the name of the script being called would help eg. sh 'script.sh' if the step was displayed as "script.sh" it would be much better than it is right now

Michael Neale added a comment - 2016-10-04 06:17

kdawgwilk good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.

Michael Neale added a comment - 2016-10-04 06:17 kdawgwilk good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.

James Dumay made changes - 2016-10-24 05:24

Rank

Ranked higher

James Dumay made changes - 2016-10-24 05:24

Rank

Ranked lower

Michael Neale added a comment - 2016-10-28 01:55

jdumay if we get labelled steps (via any means) probably can close this.

Michael Neale added a comment - 2016-10-28 01:55 jdumay if we get labelled steps (via any means) probably can close this.

James Dumay made changes - 2016-11-21 21:31

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:33

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:34

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:34

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jesse Glick added a comment - 2016-12-05 21:45

IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.

Jesse Glick added a comment - 2016-12-05 21:45 IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.

James Dumay made changes - 2016-12-06 04:10

Link

This issue relates to ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

James Dumay made changes - 2016-12-06 04:25

Link

This issue relates to ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

James Dumay added a comment - 2016-12-15 21:41 - edited

alanwu mentioned on the mailing list he would like to represent a Pipeline like the following:

Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it,
here is my two cents on how I imagine my use case should look. 

                         / Slave1 -- configure -- build -- test \
  Build and test - Slave2 -- configure -- build -- test - deploy
                         \ Slave3 -- configure -- build -- test /

James Dumay added a comment - 2016-12-15 21:41 - edited alanwu mentioned on the mailing list he would like to represent a Pipeline like the following: Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it, here is my two cents on how I imagine my use case should look. / Slave1 -- configure -- build -- test \ Build and test - Slave2 -- configure -- build -- test - deploy \ Slave3 -- configure -- build -- test /

Michael Neale added a comment - 2016-12-16 00:09

jdumay so would that could be something like:

stage('build and test') {
   parallel('Slave1' : {
       stage('configure') {
       }
       stage('build') {
       }
       stage('test') {
       }
   }
}

stage('deploy') {

}

Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.

Michael Neale added a comment - 2016-12-16 00:09 jdumay so would that could be something like: stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } } stage('deploy') { } Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.

James Dumay added a comment - 2016-12-16 03:03 - edited

michaelneale yes I'd say that syntax looks about right for what alanwu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc abayer

James Dumay added a comment - 2016-12-16 03:03 - edited michaelneale yes I'd say that syntax looks about right for what alanwu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc abayer

Patrick Redivo added a comment - 2016-12-16 14:52

Hi
I am very concerned by the same issue:
for me in BlueOcean, only stages, but all stages, should be visible with Green balls.
branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself.
Therefore, if a branch has no label, then you would see just a line (with its label).
If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess.

Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way.
I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly?
Thanks
Regards
Patrick

Patrick Redivo added a comment - 2016-12-16 14:52 Hi I am very concerned by the same issue: for me in BlueOcean, only stages, but all stages, should be visible with Green balls. branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself. Therefore, if a branch has no label, then you would see just a line (with its label). If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess. Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way. I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly? Thanks Regards Patrick

Alan Wu added a comment - 2016-12-18 19:53

Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.

Alan Wu added a comment - 2016-12-18 19:53 Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.

James Dumay added a comment - 2016-12-18 22:19

alanwu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.

James Dumay added a comment - 2016-12-18 22:19 alanwu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.

Alan Wu added a comment - 2016-12-18 23:52

jamesdumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages.

I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.

Alan Wu added a comment - 2016-12-18 23:52 jamesdumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages. I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.

James Dumay added a comment - 2016-12-19 00:00 - edited

alanwu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait

James Dumay added a comment - 2016-12-19 00:00 - edited alanwu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait

James Dumay added a comment - 2016-12-19 00:04 - edited

predivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.

James Dumay added a comment - 2016-12-19 00:04 - edited predivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.

James Dumay made changes - 2016-12-19 12:00

Link

This issue relates to ~~JENKINS-39464~~ [ ~~JENKINS-39464~~ ]

James Dumay made changes - 2016-12-28 02:41

Link

This issue relates to ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-04 01:52

Link

This issue relates to ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-04 01:53

Link

This issue is duplicated by ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-06 05:54

Rank

Ranked higher

James Dumay made changes - 2017-01-06 05:54

Rank

Ranked lower

James Dumay made changes - 2017-01-09 21:45

Remote Link

This issue links to "Jenkins/Jenkins tried to use a matrix style parallel (Web Link)" [ 15226 ]

James Nord added a comment - 2017-01-13 19:01

I thought this was the reason that stage was made into a block step?

James Nord added a comment - 2017-01-13 19:01 I thought this was the reason that stage was made into a block step?

James Dumay added a comment - 2017-01-14 02:05

teilo yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good.

James Dumay added a comment - 2017-01-14 02:05 teilo yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good .

Kuber Kaul added a comment - 2017-01-16 22:50

Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

Kuber Kaul added a comment - 2017-01-16 22:50 Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

James Dumay added a comment - 2017-01-16 22:56

kaulk as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

James Dumay added a comment - 2017-01-16 22:56 kaulk as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

James Dumay made changes - 2017-01-16 22:57

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Kuber Kaul added a comment - 2017-01-16 23:00

got it, thanks.

Kuber Kaul added a comment - 2017-01-16 23:00 got it, thanks.

Michael Neale made changes - 2017-01-19 01:20

Link

This issue blocks ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

Ian Leitch made changes - 2017-01-20 08:00

Attachment

PipelineGraphProposal.png [ 35539 ]

Ian Leitch added a comment - 2017-01-20 08:00 - edited

Proposal

This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth.

Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview.

1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept)

Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level
In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes.

2. 'stage' and 'parallel_stage' can be nested arbitrarily

The graph only shows the 1st and 2nd levels
If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design)
If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design)
Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage).
Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level
Neither can be called within a 'node' block
Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'?

3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated

stage("Lint") { node { sh "lint" }  }

stage("Build") {
  parallel_stage("Shared") { node { sh "build" } }
  parallel_stage("App 1") { node { sh "build" } }
  parallel_stage("App 2") { node { sh "build" } }
}

stage("Test") {
  parallel_stage("Shared") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
  }

  parallel_stage("App 1") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
  }

  parallel_stage("App 2") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
    parallel_stage("Functional (iPad Air 2 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPad Pro 10.2)") { node { sh "test" } }
  }
}

stage("Collect") { node { sh "collect" }  }

Ian Leitch added a comment - 2017-01-20 08:00 - edited Proposal This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth. Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview. 1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept) Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes. 2. 'stage' and 'parallel_stage' can be nested arbitrarily The graph only shows the 1st and 2nd levels If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design) If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design) Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage). Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level Neither can be called within a 'node' block Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'? 3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated stage( "Lint" ) { node { sh "lint" } } stage( "Build" ) { parallel_stage( "Shared" ) { node { sh "build" } } parallel_stage( "App 1" ) { node { sh "build" } } parallel_stage( "App 2" ) { node { sh "build" } } } stage( "Test" ) { parallel_stage( "Shared" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } } parallel_stage( "App 1" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } } parallel_stage( "App 2" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } parallel_stage( "Functional (iPad Air 2 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPad Pro 10.2)" ) { node { sh "test" } } } } stage( "Collect" ) { node { sh "collect" } }

Alan Wu added a comment - 2017-01-22 22:38

ileitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal?

stage('build and test') {
parallel('Slave1' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
parallel('Slave2' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
}

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

Alan Wu added a comment - 2017-01-22 22:38 ileitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal? stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } parallel('Slave2' : { stage('configure') { } stage('build') { } stage('test') { } } } If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

James Dumay added a comment - 2017-01-22 23:08

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

alanwu yes, that is correct - we can't display those right now.

James Dumay added a comment - 2017-01-22 23:08 If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired? alanwu yes, that is correct - we can't display those right now.

Ian Leitch added a comment - 2017-01-22 23:53

alanwu jamesdumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

Ian Leitch added a comment - 2017-01-22 23:53 alanwu jamesdumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

Alan Wu added a comment - 2017-01-22 23:58

ileitch That's great, it will be very useful to have named_step. Thank you.

Alan Wu added a comment - 2017-01-22 23:58 ileitch That's great, it will be very useful to have named_step. Thank you.

James Dumay made changes - 2017-02-08 21:21

Link

This issue is duplicated by ~~JENKINS-41781~~ [ ~~JENKINS-41781~~ ]

Ryan Hutchison made changes - 2017-02-15 18:05

Link

This issue is related to ~~JENKINS-41156~~ [ ~~JENKINS-41156~~ ]

Jesse Glick added a comment - 2017-02-15 18:47

FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

Jesse Glick added a comment - 2017-02-15 18:47 FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

Michael Neale added a comment - 2017-02-15 23:24

jglick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

Michael Neale added a comment - 2017-02-15 23:24 jglick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

James Dumay made changes - 2017-02-15 23:54

Link

This issue blocks ~~JENKINS-42062~~ [ ~~JENKINS-42062~~ ]

Michael Neale made changes - 2017-02-23 04:05

Link

This issue relates to ~~JENKINS-41205~~ [ ~~JENKINS-41205~~ ]

Michael Neale added a comment - 2017-03-20 21:21

There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by:

One level of stages shown in parallel streams of execution
some first class support for matrix of combinations (in paralllel)

I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution).

Michael Neale added a comment - 2017-03-20 21:21 There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by: One level of stages shown in parallel streams of execution some first class support for matrix of combinations (in paralllel) I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution).

Ben Dean made changes - 2017-03-21 14:43

Attachment

example.png [ 36599 ]

Ben Dean made changes - 2017-03-21 18:07

Attachment

pipeline_mockup.png [ 36602 ]

Ben Dean added a comment - 2017-03-21 18:07 - edited

Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages:

Clone SCM
Build
Build documentation on Windows slave (slow)
Custom static analysis tool (slow)
Multiple Test suites
Archive build artifacts and test results

Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO):

Jenkinsfile

stage('Clone') {
    checkout scm
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

parallel([
    buildAndTest: {
        stage('Build') {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build-artifacts'
            }
        }
        stage('Test') {
            testSteps = [:]
            for (suite in testSuites) {
                testSteps[suite] = {
                    node('linux') {
                        unstash 'build-artifacts'
                        sh "run_tests $suite"
                        stash includes: 'test_results/**/*.xml', name: "test-$suite"
                    }
                }
            }
            parallel(testSteps)
        }
    },
    docs: {
        stage('Documentation') {
            node('windows') {
                bat 'build_docs.exe'
            }
        }
    },
    staticAnalysis: {
        stage('Static Analysis') {
            node('linux') {
                sh 'run_static_analysis'
            }
        }
    }
])

stage('Archive') {
    unstash 'build-artifacts'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages.

As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render:

example.dot

digraph {
  graph [rankdir=LR]

  Clone -> Build
  Clone -> Documentation
  Clone -> "Static Analysis"

  Documentation -> Archive
  "Static Analysis" -> Archive

  Build -> "Test suite1"
  Build -> "Test suite2"
  Build -> "Test suite3"

  "Test suite1" -> Archive
  "Test suite2" -> Archive
  "Test suite3" -> Archive
}

Here's what I was thinking it might look like in Blue Ocean

The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this:

Jenkinsfile

stage('Clone') {
    checkout scm
}

stage('Build') {
    parallel([
        build: {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build'
            }
        },
        docs: {
            build job: 'docs', wait: false
        },
        staticAnalysis: {
            build job: 'staticAnalysis', wait: false
        }
    ])
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

stage('Test') {
    testSteps = [:]
    for (suite in testSuites) {
        testSteps[suite] = {
            node('linux') {
                unstash 'build'
                sh "run_tests $suite"
                stash includes: 'test_results/**/*.xml', name: "test-$suite"
            }
        }
    }
    parallel(testSteps)
}

// method to lookup the docs and staticAnalysis jobs we started and see if they're done
@NonCPS
def findSomeBuild(jobName)
    // do Jenkins groovy stuff
    return correctBuildNumber || null
end

stage('Archive') {
    unstash 'build'

    def docsBuildNumber = null    
    waitUntil {
        docsBuildNumber = findSomeBuild('docs')
    }
    
    def staticAnalysisBuildNumber = null
    waitUntil {
        staticAnalysisBuildNumber = findSomeBuild('staticAnalysis')
    }
    
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'docs', selector: [$class: 'SpecificBuildSelector', buildNumber: docsBuildNumber]])
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'staticAnalysis', selector: [$class: 'SpecificBuildSelector', buildNumber: staticAnalysisBuildNumber]])

    unstash 'documentation'
    unstash 'static_analysis'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

Ben Dean added a comment - 2017-03-21 18:07 - edited Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages: Clone SCM Build Build documentation on Windows slave (slow) Custom static analysis tool (slow) Multiple Test suites Archive build artifacts and test results Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO): Jenkinsfile stage( 'Clone' ) { checkout scm } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] parallel([ buildAndTest: { stage( 'Build' ) { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build-artifacts' } } stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build-artifacts' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } }, docs: { stage( 'Documentation' ) { node( 'windows' ) { bat 'build_docs.exe' } } }, staticAnalysis: { stage( 'Static Analysis' ) { node( 'linux' ) { sh 'run_static_analysis' } } } ]) stage( 'Archive' ) { unstash 'build-artifacts' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' } I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages. As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render: example.dot digraph { graph [rankdir=LR] Clone -> Build Clone -> Documentation Clone -> "Static Analysis" Documentation -> Archive "Static Analysis" -> Archive Build -> "Test suite1" Build -> "Test suite2" Build -> "Test suite3" "Test suite1" -> Archive "Test suite2" -> Archive "Test suite3" -> Archive } Here's what I was thinking it might look like in Blue Ocean The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this: Jenkinsfile stage( 'Clone' ) { checkout scm } stage( 'Build' ) { parallel([ build: { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build' } }, docs: { build job: 'docs' , wait: false }, staticAnalysis: { build job: 'staticAnalysis' , wait: false } ]) } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } // method to lookup the docs and staticAnalysis jobs we started and see if they're done @NonCPS def findSomeBuild(jobName) // do Jenkins groovy stuff return correctBuildNumber || null end stage( 'Archive' ) { unstash 'build' def docsBuildNumber = null waitUntil { docsBuildNumber = findSomeBuild( 'docs' ) } def staticAnalysisBuildNumber = null waitUntil { staticAnalysisBuildNumber = findSomeBuild( 'staticAnalysis' ) } step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'docs' , selector: [$class: 'SpecificBuildSelector' , buildNumber: docsBuildNumber]]) step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'staticAnalysis' , selector: [$class: 'SpecificBuildSelector' , buildNumber: staticAnalysisBuildNumber]]) unstash 'documentation' unstash 'static_analysis' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' }

Laurent Gillet added a comment - 2017-03-21 18:53 - edited

@b_dean: I totally agree with your proposition!

We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

Laurent Gillet added a comment - 2017-03-21 18:53 - edited @ b_dean : I totally agree with your proposition! We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

Andrew Miller added a comment - 2017-03-28 00:15

May I suggest the following as a means to represent nested stages visually?

Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts.

This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic.

I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

Andrew Miller added a comment - 2017-03-28 00:15 May I suggest the following as a means to represent nested stages visually? Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts. This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic. I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

James Dumay added a comment - 2017-03-28 01:34

ikari7789 could you draw a picture and upload it here please?

James Dumay added a comment - 2017-03-28 01:34 ikari7789 could you draw a picture and upload it here please?

Andrew Miller made changes - 2017-04-12 07:16

Attachment

image-2017-04-12-16-16-17-144.png [ 37113 ]

Andrew Miller made changes - 2017-04-12 07:17

Attachment

image-2017-04-12-16-17-38-285.png [ 37114 ]

Andrew Miller added a comment - 2017-04-12 07:19

Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind.

So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final).

When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment):

This process could basically allow for an infinite expansion inwards.

The red/green status would be determined by the final result of all children stages.

Andrew Miller added a comment - 2017-04-12 07:19 Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind. So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final). When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment): This process could basically allow for an infinite expansion inwards. The red/green status would be determined by the final result of all children stages.

Prateek VERMA added a comment - 2017-04-12 08:35

Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise.

In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps.

Lable foo
---O--O--O-----------

s1 s2 s3

Lable bar

---O--O--O-
s1 s2 s3

Prateek VERMA added a comment - 2017-04-12 08:35 Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise. In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps. Lable foo --- O -- O -- O ----------- s1 s2 s3 Lable bar --- O -- O -- O - s1 s2 s3

Prateek VERMA made changes - 2017-04-12 08:35

Assignee

Jesse Glick [ jglick ]

Michael Neale made changes - 2017-04-12 08:37

Assignee

Jesse Glick [ jglick ]

Prateek VERMA added a comment - 2017-04-12 08:39

Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

Prateek VERMA added a comment - 2017-04-12 08:39 Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

Prateek VERMA made changes - 2017-04-12 08:53

Attachment

Proposed Parallel Stages.JPG [ 37115 ]

Prateek VERMA added a comment - 2017-04-12 08:53

Please refer to "Proposed Parallel Stages.JPG" attached here.

Prateek VERMA added a comment - 2017-04-12 08:53 Please refer to "Proposed Parallel Stages.JPG" attached here.

ben powers made changes - 2017-04-12 18:59

Attachment

alternate zoom in.png [ 37120 ]

ben powers added a comment - 2017-04-12 19:06

I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources

As he mentioned, this should provide a UI framework to support additional nesting.

Here's a sketch that might help Andrew feel better about his artwork Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

ben powers added a comment - 2017-04-12 19:06 I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources As he mentioned, this should provide a UI framework to support additional nesting. Here's a sketch that might help Andrew feel better about his artwork Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

Tushar Mahale made changes - 2017-04-14 01:26

Assignee

Tushar Mahale [ tusharmahale ]

James Dumay made changes - 2017-04-14 01:32

Assignee

Tushar Mahale [ tusharmahale ]

Jean Mertz added a comment - 2017-04-27 15:31

I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations.

It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

Jean Mertz added a comment - 2017-04-27 15:31 I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations. It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

James Dumay made changes - 2017-05-01 04:30

Epic Link

JENKINS-35750 [ 171713 ]

JENKINS-43953 [ 181485 ]

James Dumay made changes - 2017-06-19 01:56

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2017-06-19 01:56

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2017-06-19 01:57

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jean-Paul G made changes - 2017-06-22 04:20

Link

This issue is related to ~~JENKINS-44820~~ [ ~~JENKINS-44820~~ ]

James Dumay made changes - 2017-08-13 05:48

Link

This issue is duplicated by ~~JENKINS-45664~~ [ ~~JENKINS-45664~~ ]

Joerg Schwaerzler made changes - 2017-08-14 08:29

Attachment

image-2017-08-14-10-29-55-373.png [ 39258 ]

Joerg Schwaerzler made changes - 2017-08-14 08:34

Attachment

image-2017-08-14-10-34-10-452.png [ 39259 ]

Joerg Schwaerzler added a comment - 2017-08-14 08:36 - edited

How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.:

stage ('Build') {
    // Build something
}
parallel('Test1' : {
    stage('JUnit') {
    }
    stage('Firefox') {
    }
}, 'Test2' : {
    stage('DBUnit') {
    }
    stage('Edge') {
    }
}, 'Test3' : {
    stage('Jasmine') {
    }
    stage('Safari') {
    }    
}
stage ('Dev') {
}

Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

Joerg Schwaerzler added a comment - 2017-08-14 08:36 - edited How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.: stage ( 'Build' ) { // Build something } parallel( 'Test1' : { stage( 'JUnit' ) { } stage( 'Firefox' ) { } }, 'Test2' : { stage( 'DBUnit' ) { } stage( 'Edge' ) { } }, 'Test3' : { stage( 'Jasmine' ) { } stage( 'Safari' ) { } } stage ( 'Dev' ) { } Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

Michael Neale added a comment - 2017-08-14 08:37

macdrega that I think is what most people agree on as some kind of next step for this.

Michael Neale added a comment - 2017-08-14 08:37 macdrega that I think is what most people agree on as some kind of next step for this.

Joerg Schwaerzler added a comment - 2017-08-14 08:40

Great than, was not sure about that.

Joerg Schwaerzler added a comment - 2017-08-14 08:40 Great than, was not sure about that.

Joerg Schwaerzler made changes - 2017-08-14 08:41

Attachment

image-2017-08-14-10-29-55-373.png [ 39258 ]

Florian Manschwetus added a comment - 2017-08-14 08:43 - edited

macdrega +1, no more words on this needed from my side.

Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

Florian Manschwetus added a comment - 2017-08-14 08:43 - edited macdrega +1, no more words on this needed from my side. Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

Joerg Schwaerzler added a comment - 2017-08-14 09:00

manschwetus

Sorry for that - just wanted to be crystal clear about what I think should be the next step.

Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts.

But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks

However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

Joerg Schwaerzler added a comment - 2017-08-14 09:00 manschwetus Sorry for that - just wanted to be crystal clear about what I think should be the next step. Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts. But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

James Dumay made changes - 2017-08-28 17:39

Link

This issue is duplicated by ~~JENKINS-46345~~ [ ~~JENKINS-46345~~ ]

Michael Neale made changes - 2017-10-12 01:59

Remote Link

This issue links to "Page (Jenkins Wiki)" [ 17861 ]

Henri Anttila made changes - 2017-10-19 05:22

Attachment

screenshot-1.png [ 40062 ]

Henri Anttila made changes - 2017-10-19 05:24

Attachment

image-2017-10-19-08-24-00-504.png [ 40063 ]

Henri Anttila added a comment - 2017-10-19 05:25

Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so:

These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

Henri Anttila added a comment - 2017-10-19 05:25 Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so: These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

Michael Neale added a comment - 2017-10-19 05:27

xtrom0rt do you mean more like a matrix of combinations?

Michael Neale added a comment - 2017-10-19 05:27 xtrom0rt do you mean more like a matrix of combinations?

Josh McDonald added a comment - 2017-10-19 07:22

Maybe we should start thinking about a BO-style renderer for matrix jobs?

Josh McDonald added a comment - 2017-10-19 07:22 Maybe we should start thinking about a BO-style renderer for matrix jobs?

Michael Neale added a comment - 2017-10-19 07:27

sophistifunk yes - and I know abayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed.

Michael Neale added a comment - 2017-10-19 07:27 sophistifunk yes - and I know abayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed.

Henri Anttila added a comment - 2017-10-19 09:30

I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization.

For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary.

After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

Henri Anttila added a comment - 2017-10-19 09:30 I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization. For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary. After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

Chris Majoros added a comment - 2017-10-19 17:24

I am actually looking to do the same thing as xtromort

Chris Majoros added a comment - 2017-10-19 17:24 I am actually looking to do the same thing as xtromort

Mathew Joseph added a comment - 2017-11-19 20:43

Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

Mathew Joseph added a comment - 2017-11-19 20:43 Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

Michael Neale added a comment - 2017-11-19 22:44

cryptomatt not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet.

Michael Neale added a comment - 2017-11-19 22:44 cryptomatt not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet.

James Dumay made changes - 2017-11-30 23:23

Remote Link

This issue links to "CloudBees Internal UX-584 (Web Link)" [ 18213 ]

Nicholas Roth added a comment - 2018-01-02 19:18

This would be really nice to have!

Nicholas Roth added a comment - 2018-01-02 19:18 This would be really nice to have!

Roy Arnon added a comment - 2018-01-03 15:04 - edited

+1 from me as well.

In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules. We are currently converting them to pipelines, and we parallelize a lot of steps:

Build Jars
Package & Test (Parallel)
1. Build RPM
2. Build Docker
3. Run Tests (Parallel)
Promote build

We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps.

Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps.

I also think the ikari7789 suggestion seems like the best option.

Roy Arnon added a comment - 2018-01-03 15:04 - edited +1 from me as well. In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules. We are currently converting them to pipelines, and we parallelize a lot of steps: Build Jars Package & Test (Parallel) Build RPM Build Docker Run Tests (Parallel) Promote build We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps. Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps. I also think the ikari7789 suggestion seems like the best option.

Florian Wilhelm added a comment - 2018-01-08 16:12

I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved.

I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future?

Best regards,
Florian

Florian Wilhelm added a comment - 2018-01-08 16:12 I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved. I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future? Best regards, Florian

Gabriel Loewen made changes - 2018-03-09 16:10

Attachment

Before.PNG [ 41786 ]

Gabriel Loewen made changes - 2018-03-09 16:10

Attachment

After.png [ 41787 ]

Gabriel Loewen added a comment - 2018-03-09 16:14

Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following:

pipeline {
    agent any
    stages {
        stage("Testing") {
            steps {
                script {
                    stage("A") {
                        parallel "1": {
                            println "1"
			    sleep 500
                        }, "2": {
                            println "2"
			    sleep 500
                        }, "3": {
                            println "3"
			    sleep 500
                        }
                    }
                }
            }
        }
    }
}

Then while this pipeline is running I see the following in Blue Ocean:

But after the pipeline finishes the nested stage and the parent stage get merged:

What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

Gabriel Loewen added a comment - 2018-03-09 16:14 Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following: pipeline { agent any stages { stage( "Testing" ) { steps { script { stage( "A" ) { parallel "1" : { println "1" sleep 500 }, "2" : { println "2" sleep 500 }, "3" : { println "3" sleep 500 } } } } } } } Then while this pipeline is running I see the following in Blue Ocean: But after the pipeline finishes the nested stage and the parent stage get merged: What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

Costin Caraivan added a comment - 2018-03-15 12:06 - edited

This is quite important for many CI scenarios. Quick example:

first stage: build - executed on a build node
parallel launch of 3 logical "blocks":
___ 1 block for 1 type of integration test - executed on a specialized integration node

___ 1 block for 1 other type of integration test - executed on another specialized integration node
___ 1 bigger block for deployment and functional tests:
___1 stage for deployment - executed on the target app environment
___1 stage for functional tests - executed on a specialized functional test runner node

Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless.

In my opinion perfect is the enemy of good.

Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough.

This should simplify the implementation of both the actual pipeline code and also the visualization.

Costin Caraivan added a comment - 2018-03-15 12:06 - edited This is quite important for many CI scenarios. Quick example: first stage: build - executed on a build node parallel launch of 3 logical "blocks": ___ 1 block for 1 type of integration test - executed on a specialized integration node ___ 1 block for 1 other type of integration test - executed on another specialized integration node ___ 1 bigger block for deployment and functional tests: ___1 stage for deployment - executed on the target app environment ___1 stage for functional tests - executed on a specialized functional test runner node Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless. In my opinion perfect is the enemy of good. Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough. This should simplify the implementation of both the actual pipeline code and also the visualization.

Anthony BOUQUET added a comment - 2018-03-20 11:00

+1 for ccaraivan

Have long running step and can't execute it in a parallel way because of this nested pipeline limitation.

3 level of nested pipeline should be enought

Anthony BOUQUET added a comment - 2018-03-20 11:00 +1 for ccaraivan Have long running step and can't execute it in a parallel way because of this nested pipeline limitation. 3 level of nested pipeline should be enought

Hokwang Lee added a comment - 2018-03-28 13:57 - edited

+1

I think this is really really needed for making flexible pipeline.

Because of feature toggling,

we need

checkout - build - test

... (so many)

(only different is source code checkouted and build command line)

If this feature can not be implemented, I will search another tool.

Hokwang Lee added a comment - 2018-03-28 13:57 - edited +1 I think this is really really needed for making flexible pipeline. Because of feature toggling, we need checkout - build - test checkout - build - test checkout - build - test ... (so many) (only different is source code checkouted and build command line) If this feature can not be implemented, I will search another tool.

Todd Densmore added a comment - 2018-06-20 21:36

Intel is also very interested in this visualization enhancement.

Todd Densmore added a comment - 2018-06-20 21:36 Intel is also very interested in this visualization enhancement.

Jenn Briden added a comment - 2018-06-20 23:03

We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227.

Jenn Briden added a comment - 2018-06-20 23:03 We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227.

Jenn Briden made changes - 2018-06-20 23:49

Assignee

CloudBees Inc. [ cloudbees ]

Joerg Schwaerzler added a comment - 2018-06-21 06:09

Great to hear

Two questions:

Are you also planning to support the scripted pipeline?
As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.

Joerg Schwaerzler added a comment - 2018-06-21 06:09 Great to hear Two questions: Are you also planning to support the scripted pipeline? As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.

Jenn Briden made changes - 2018-07-06 21:15

Status

Open [ 1 ]

In Progress [ 3 ]

Jenn Briden made changes - 2018-07-06 21:18

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}

There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}
*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jenn Briden made changes - 2018-07-06 21:19

Description

{panel}

There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}
*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}
In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jenn Briden made changes - 2018-07-06 21:20

Summary

Support nested stages in the Pipeline Graph

View sequential stages in the pipeline visualization graph

Domagoj Smoljanovic added a comment - 2018-08-01 09:04

I see that the above PR is merged but issue is still in progress. Is this still being worked on?

When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that.

Thanx

Domagoj Smoljanovic added a comment - 2018-08-01 09:04 I see that the above PR is merged but issue is still in progress. Is this still being worked on? When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that. Thanx

Josh McDonald added a comment - 2018-08-01 09:14

AFAIK there's no plan to support nested parallels at the moment.

Josh McDonald added a comment - 2018-08-01 09:14 AFAIK there's no plan to support nested parallels at the moment.

Daniel Kurzynski added a comment - 2018-08-01 09:21

We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

Daniel Kurzynski added a comment - 2018-08-01 09:21 We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

Mathias Hasselmann added a comment - 2018-08-01 10:24

I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

Mathias Hasselmann added a comment - 2018-08-01 10:24 I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

Gert van Dijk made changes - 2018-08-14 08:27

Attachment

jenkins-weird-masked2.png [ 43737 ]

Gert van Dijk added a comment - 2018-08-14 08:33 - edited

First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway.

However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent.

It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage. In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages.

Sub-stages are shown during the run, though.

As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues.
(Seeing this on 1.7.2 / Core 2.121.2)

Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

Gert van Dijk added a comment - 2018-08-14 08:33 - edited First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway. However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent. It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage . In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages. Sub-stages are shown during the run , though. As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues. (Seeing this on 1.7.2 / Core 2.121.2) Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

Devin Nusbaum added a comment - 2018-08-17 19:13 - edited

Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of ~~JENKINS-49050~~ (although note that I believe that issue is specific to declarative pipeline).

Devin Nusbaum added a comment - 2018-08-17 19:13 - edited Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of JENKINS-49050 (although note that I believe that issue is specific to declarative pipeline).

Devin Nusbaum made changes - 2018-08-17 19:13

Link

This issue duplicates ~~JENKINS-49050~~ [ ~~JENKINS-49050~~ ]

Devin Nusbaum made changes - 2018-08-17 19:13

Resolution		Duplicate [ 3 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Devin Nusbaum made changes - 2018-08-17 19:14

Released As

Blue Ocean 1.8.0

Joerg Schwaerzler added a comment - 2018-08-18 05:37

dnusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?

Joerg Schwaerzler added a comment - 2018-08-18 05:37 dnusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?

Devin Nusbaum added a comment - 2018-08-20 13:12

macdrega I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

Devin Nusbaum added a comment - 2018-08-20 13:12 macdrega I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

Daniel Kurzynski added a comment - 2018-08-20 13:20 - edited

I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization.
Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

Daniel Kurzynski added a comment - 2018-08-20 13:20 - edited I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization. Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

Devin Nusbaum added a comment - 2018-08-20 13:24

kurzy Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

Devin Nusbaum added a comment - 2018-08-20 13:24 kurzy Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

Joerg Schwaerzler added a comment - 2018-08-21 12:28

dnusbaum: Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

Joerg Schwaerzler added a comment - 2018-08-21 12:28 dnusbaum : Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

Joerg Schwaerzler added a comment - 2018-08-21 12:41

For those interested: I raised a new fr here: ~~JENKINS-53162~~

Joerg Schwaerzler added a comment - 2018-08-21 12:41 For those interested: I raised a new fr here: JENKINS-53162

Devin Nusbaum made changes - 2018-08-21 13:36

Link

This issue relates to ~~JENKINS-53162~~ [ ~~JENKINS-53162~~ ]

Phil Segal added a comment - 2018-10-10 09:35

sophistifunk is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN . Something like pipeline_mockups.png from above would be amazing

Phil Segal added a comment - 2018-10-10 09:35 sophistifunk is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN . Something like pipeline_mockups.png from above would be amazing

Daniel Kurzynski added a comment - 2018-10-11 08:52

dnusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

Daniel Kurzynski added a comment - 2018-10-11 08:52 dnusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

Jesse Glick made changes - 2018-10-11 14:04

Link

This issue relates to JENKINS-54010 [ JENKINS-54010 ]

zengqiao liu made changes - 2018-12-17 20:59

Description

{panel}
In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}

In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.
{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Ahmed Sghaier added a comment - 2019-01-11 13:02

Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647

Should we create a separate issue for this?

Ahmed Sghaier added a comment - 2019-01-11 13:02 Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647 Should we create a separate issue for this?

Anentropic made changes - 2020-11-26 17:15

Attachment

current.png [ 53474 ]

Anentropic made changes - 2020-11-26 17:16

Attachment

Jenkins-substages.png [ 53475 ]

Anentropic added a comment - 2020-11-26 17:17

This is a very long thread so forgive me if this is already covered...

Can anyone explain why a stage that contains sub-stages is displayed as a preceding node instead of a container?

Take this example:

The stage "Test" performs no actions of its own, but it contains the sub-stages that follow.

Wouldn't it make more sense to display it as a container, something like:

Anentropic added a comment - 2020-11-26 17:17 This is a very long thread so forgive me if this is already covered... Can anyone explain why a stage that contains sub-stages is displayed as a preceding node instead of a container? Take this example: The stage "Test" performs no actions of its own, but it contains the sub-stages that follow. Wouldn't it make more sense to display it as a container, something like:

Jenkins

View sequential stages in the pipeline visualization graph

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates