[JENKINS-38442] View sequential stages in the pipeline visualization graph - Jenkins Jira

mohamed badran created issue - 2016-09-22 11:03

Hide

Permalink

Michael Neale added a comment - 2016-09-22 22:14

Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph).

For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?

Show

Michael Neale added a comment - 2016-09-22 22:14 Ah, is this is nested stages, they won't be supported visually for some time (there is some question on how to show them, labelled steps below the parent stage, or a nested block of stages in the graph). For now, don't use nested stages, it won't do much good. I am curious what your aim is - you want to label steps or blocks of steps as doing something significant?

James Dumay made changes - 2016-09-23 03:31

Field	Original Value	New Value
Issue Type	Bug [ 1 ]	Story [ 10002 ]

James Dumay made changes - 2016-09-23 03:31

Summary

Nested labelled blocks breaks blueocean

Support nested stages in the Pipeline Graph

James Dumay made changes - 2016-09-23 03:33

Description

I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:34

Description

{panel title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=blue}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=grey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:35

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:45

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:46

Description

{panel:title="Support for nested stages is currently unplanned"|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

James Dumay made changes - 2016-09-23 03:46

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

Hide

Permalink

James Dumay added a comment - 2016-09-23 06:24

mohamed badran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?

Show

James Dumay added a comment - 2016-09-23 06:24 mohamed badran thanks for reporting this. Do you have a concrete use case you could share with us for nested stages?

Hide

Permalink

mohamed badran added a comment - 2016-09-25 09:06

Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script.

Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.

Show

mohamed badran added a comment - 2016-09-25 09:06 Well currently I'm using it to be able to be able to have a like a container for the shell script in order to know what this shell script is doing since i'm not able to add a description to the shell script that will show up next to Shell Script in blueocean, as well as pipeline visualization. My ideal solution in here is to have availability to add description next to shell script so if i'm running multiple shell scripts in one stage i'd be able to easily know what is that shell script doing. Currently there are no way to do it as i understand so I've tried using nested stages to be able to separate the shell scripts and know what they do. Usage of nested stages like this is a workaround for lack of ability to add description for the shell script. Regarding a concrete use case of using nested stages would be being able to relate multiple stages to one stage, for example if I've a major step which prepares the environment, and that preparation consists of fewer stages for example 1- cleaning old data, 2- copy new data, 3- prepare area to run builds. My ideal look for that would be a parent stage which is called Prepare environment, and then when pressing on that node it would fork 3 other nodes (which are the 3 stages I've mentioned), and when you select any of the inner stages you'll be able to get the shell scripts and any other pipeline commands you were doing in such stage.

Hide

Permalink

Michael Neale added a comment - 2016-09-25 22:53

mohamed badran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?)

Perhaps we need to draw out what the options could be for this.

But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?

Show

Michael Neale added a comment - 2016-09-25 22:53 mohamed badran so for now it shows up as a log prefix which helps you a bit - but ideally it would show up as a labelled step (or group of scripts, that are labelled as one thing you can "expand" into?) Perhaps we need to draw out what the options could be for this. But to me the main thing is you need to label a step, or group together a bunch of steps under a label... I think? Which implies that you really want stages to be the top level things, and when you drill down into them, you see groupings of labelled steps - woudl that be fair to say?

James Dumay made changes - 2016-09-26 06:21

Epic Link

JENKINS-35750 [ 171713 ]

Hide

Permalink

mohamed badran added a comment - 2016-09-26 08:53

Michael Neale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing.

In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.

Show

mohamed badran added a comment - 2016-09-26 08:53 Michael Neale You got it correctly Michael that i need to have label to a step (if possible we can make that label applicable on any kind of step not just a shell script), or group together bunch of steps under a label. This implies what you've mentioned as well that i need to have a top level stage that i can drill down into it to see grouping of labeled steps, or i see in the log a step next to it a label in order to know what is that step doing. In my real use case I could have for example like 16 shell scripts running under one stage in parallel to build variant platforms, looking at this in blueocean, or pipeline view would show me 16 shell scripts and that's it which makes it pretty hard for developers to know which build actually failed, unless they go to pipeline steps to know the stage name and they figure out which build actually failed, or they've to do a full log view in order to find out the command that was used for a build.

Hide

Permalink

Michael Neale added a comment - 2016-09-27 05:44 - edited

mohamed badran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit.

But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see...

James Dumay perhaps this is a case for labelled steps of some kind?

Show

Michael Neale added a comment - 2016-09-27 05:44 - edited mohamed badran well, some slight good news: there is a proposal that more information about scripts is returned so instead of saying "shell script" it can show a snippet of the script itself (just a little bit) or the output if run (that isn't being done as part of blue ocean project though, as deeper changes are needed) - so that helps a bit. But to clarify, this is clearer. So you are happy with a stage graph at a high level, up the top, but need ways to group multiple steps into one, and label them (down the bottom). Maybe nested stage is useful for that, but not sure. To me nested stage implies a nesting of the visual graph, but we will see... James Dumay perhaps this is a case for labelled steps of some kind?

mohamed badran made changes - 2016-09-27 09:21

Attachment

Capture.JPG [ 34084 ]

Hide

Permalink

mohamed badran added a comment - 2016-09-27 09:47

Michael Neale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage.

For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages).

Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this

steps a/ab
shell script

steps a/xyz
shell script

These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like
shell script : (This is platform 1)
shell script: (This is platform 2)

So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.

Show

mohamed badran added a comment - 2016-09-27 09:47 Michael Neale The stage graph i think should be enhanced as well to give any indication that the the node actually is formed from multi-stages for example you can have a node with double circles around it to give a visual indicator that something is different in this stage. For example in the code I've given earlier if you actually uncomment //stage 'start' it will show in the graph something like this if node "a" and node "b" have double rings around them then this could mean to the user that something is different in that stage. I think to support nested stages maybe when clicking on a node that is formed of multistage you can remove the original high level graph and just zoom in the small graph only so in my case when i press on node "a" that would remove the original graph, and show new graph of "a" as start then "ab", and "xyz" as children for it, and if we've further down stages you can always clean the visualization and show just all the previous nodes as a line formed having multi nodes in there, and only the deepest stage you should start to expand it. Once the user finishes there could be a reset button to reset the graphs to the parent, or they just press on first node. (This is just an proposal, but the simplest would be just a different indicator for a node that is formed of multi stages). Regarding the bottom part I'd like in the attached image to see the first shell script under a/ab, and the second shell script under a/xyz so it could be something like this steps a/ab shell script steps a/xyz shell script These are all thoughts to try to help in how to visualize things but definitely there could be other ways. To me for example i'd want something like shell script : (This is platform 1) shell script: (This is platform 2) So yes i need something either in the visualization part, or in the lower level that would show that nesting in someway and show the grouping, and preferably both parts. But i just need to give me any means of feedback of what's happening.

James Dumay made changes - 2016-09-28 05:04

Attachment

blueocean.sketch 2016-09-28 15-03-57.png [ 34100 ]

Hide

Permalink

James Dumay added a comment - 2016-09-28 05:06

One way we could make this work is:

Where we have a Pipeline like

  stage ("Build") {
    stage ("Inner 1") {
       ....
    }
    stage ("Inner 2") {
        ....
    }
  }

Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.

Show

James Dumay added a comment - 2016-09-28 05:06 One way we could make this work is: Where we have a Pipeline like stage ( "Build" ) { stage ( "Inner 1" ) { .... } stage ( "Inner 2" ) { .... } } Then again I've heard use cases where these become boxes around stages in the pipeline graph or lines connecting stages together within stages.... need to go back to the Pipeline developers and get guidance around what the intended use case was here.

Hide

Permalink

mohamed badran added a comment - 2016-09-28 09:11

James Dumay and Michael Neale

Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.

Show

mohamed badran added a comment - 2016-09-28 09:11 James Dumay and Michael Neale Actually that last comment image looks good, and it actually follows one of my suggestions as well so that will be good enhancement. One point as you mentioned in the upper graph to add like any visualization indicator that this is a nested stage as a box, or as anything else.

Hide

Permalink

James Dumay added a comment - 2016-09-29 00:06

Im still hesitant because we have no idea what pipelines like this should look like

stage ("outer") {
    stage ("inner") {
       stage ("most inner") {
          stage ("turtles all the way down") {
             ...
          }
       }
    }
  }

Show

James Dumay added a comment - 2016-09-29 00:06 Im still hesitant because we have no idea what pipelines like this should look like stage ( " outer " ) { stage ( " inner " ) { stage ( "most inner " ) { stage ( "turtles all the way down" ) { ... } } } }

Hide

Permalink

Michael Neale added a comment - 2016-09-29 00:28 - edited

Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config.

I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.

Show

Michael Neale added a comment - 2016-09-29 00:28 - edited Right - are there nested stages in a visual stage graph, or are they grouping of steps? if the latter, then there is an assumption that nested stages are represented that way that isn't obvious from the config. I don't think anyone has clearly asked for nested stages (given no one can explain what they are meant to be). What has been asked for is a way to label a step, or group steps under a label. nested stages does not achieve this clearly.

Hide

Permalink

mohamed badran added a comment - 2016-09-29 09:11

Michael Neale

Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything.

James Dumay

Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like:

Steps Outer
Inner Stage inner
Inner Stage inner/most inner

I believe by this way users will always be able to see things clearly

Show

mohamed badran added a comment - 2016-09-29 09:11 Michael Neale Nested stages doesn't do the the label part but it is a workaround, however nested stages could be needed in cases as I've mentioned earlier. Visual graph(I'm talking about Cloudbees one not blueocean upper graph) doesn't understand any nesting each stage will always show each stage as a separate block which is really ugly since you never know that this step is related to anything. James Dumay Having nested stages could still be expanded in a flat way. This could happen the same way you did it previously but add like a path to it so that would look like: Steps Outer Inner Stage inner Inner Stage inner/most inner I believe by this way users will always be able to see things clearly

Kaden Wilkinson made changes - 2016-10-03 22:14

Attachment

Screen Shot 2016-10-03 at 4.13.46 PM.png [ 34227 ]

Hide

Permalink

Kaden Wilkinson added a comment - 2016-10-03 22:15

I think the reason many are wanting to even have nested stages is to help solve this problem:

We would like to give these shell scripts a more descriptive name but are unable to.

Show

Kaden Wilkinson added a comment - 2016-10-03 22:15 I think the reason many are wanting to even have nested stages is to help solve this problem: We would like to give these shell scripts a more descriptive name but are unable to.

Hide

Permalink

Michael Neale added a comment - 2016-10-03 23:11 - edited

Kaden Wilkinson exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?

Show

Michael Neale added a comment - 2016-10-03 23:11 - edited Kaden Wilkinson exactly - that is exactly what I was thinking. Do you want to label those steps explicitly, or if they showed a more helpful description (say, the first N chars of the script) would that help?

Hide

Permalink

Kaden Wilkinson added a comment - 2016-10-03 23:25 - edited

I would like to have a syntax in the Jenkinfile to be able to say:

sh([cmd: 'script.sh', name: 'Deploy script'])

or maybe something more generic that would work for all build steps?

step('Deploy script') {
    sh 'script.sh'
}

Show

Kaden Wilkinson added a comment - 2016-10-03 23:25 - edited I would like to have a syntax in the Jenkinfile to be able to say: sh([cmd: 'script.sh' , name: 'Deploy script' ]) or maybe something more generic that would work for all build steps? step( 'Deploy script' ) { sh 'script.sh' }

Hide

Permalink

Kaden Wilkinson added a comment - 2016-10-03 23:28

Even the name of the script being called would help eg.

sh 'script.sh'

if the step was displayed as "script.sh" it would be much better than it is right now

Show

Kaden Wilkinson added a comment - 2016-10-03 23:28 Even the name of the script being called would help eg. sh 'script.sh' if the step was displayed as "script.sh" it would be much better than it is right now

Hide

Permalink

Michael Neale added a comment - 2016-10-04 06:17

Kaden Wilkinson good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.

Show

Michael Neale added a comment - 2016-10-04 06:17 Kaden Wilkinson good to know. Yes showing something meaningful certainly should happen no matter what. Labelling is also very nice.

James Dumay made changes - 2016-10-24 05:24

Rank

Ranked higher

James Dumay made changes - 2016-10-24 05:24

Rank

Ranked lower

Hide

Permalink

Michael Neale added a comment - 2016-10-28 01:55

James Dumay if we get labelled steps (via any means) probably can close this.

Show

Michael Neale added a comment - 2016-10-28 01:55 James Dumay if we get labelled steps (via any means) probably can close this.

James Dumay made changes - 2016-11-21 21:31

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

Code used:

{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:33

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:34

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {stage} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2016-11-21 21:34

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Hide

Permalink

Jesse Glick added a comment - 2016-12-05 21:45

IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.

Show

Jesse Glick added a comment - 2016-12-05 21:45 IMO you should stop showing individual steps at all (display the complete output per block), and just let stage be used to show substructure in the natural way.

James Dumay made changes - 2016-12-06 04:10

Link

This issue relates to ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

James Dumay made changes - 2016-12-06 04:25

Link

This issue relates to ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

Hide

Permalink

James Dumay added a comment - 2016-12-15 21:41 - edited

Alan Wu mentioned on the mailing list he would like to represent a Pipeline like the following:

Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it,
here is my two cents on how I imagine my use case should look. 

                         / Slave1 -- configure -- build -- test \
  Build and test - Slave2 -- configure -- build -- test - deploy
                         \ Slave3 -- configure -- build -- test /

Show

James Dumay added a comment - 2016-12-15 21:41 - edited Alan Wu mentioned on the mailing list he would like to represent a Pipeline like the following: Says if I now have the parallel block within a stage called "Build and test" and add in a "deploy" stage after it, here is my two cents on how I imagine my use case should look. / Slave1 -- configure -- build -- test \ Build and test - Slave2 -- configure -- build -- test - deploy \ Slave3 -- configure -- build -- test /

Hide

Permalink

Michael Neale added a comment - 2016-12-16 00:09

James Dumay so would that could be something like:

stage('build and test') {
   parallel('Slave1' : {
       stage('configure') {
       }
       stage('build') {
       }
       stage('test') {
       }
   }
}

stage('deploy') {

}

Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.

Show

Michael Neale added a comment - 2016-12-16 00:09 James Dumay so would that could be something like: stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } } stage('deploy') { } Given that everything needs to happen under a parent stage. The nested stages in parallel are the different groups of steps for configure etc.

Hide

Permalink

James Dumay added a comment - 2016-12-16 03:03 - edited

Michael Neale yes I'd say that syntax looks about right for what Alan Wu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc Andrew Bayer

Show

James Dumay added a comment - 2016-12-16 03:03 - edited Michael Neale yes I'd say that syntax looks about right for what Alan Wu is proposing. Though it gets ugly with multiple parallel branches as you have to repeat the stage structure. Perhaps we need to come up with a specialised syntax to support this structure? Worth exploring IMO cc Andrew Bayer

Hide

Permalink

Patrick Redivo added a comment - 2016-12-16 14:52

Hi
I am very concerned by the same issue:
for me in BlueOcean, only stages, but all stages, should be visible with Green balls.
branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself.
Therefore, if a branch has no label, then you would see just a line (with its label).
If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess.

Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way.
I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly?
Thanks
Regards
Patrick

Show

Patrick Redivo added a comment - 2016-12-16 14:52 Hi I am very concerned by the same issue: for me in BlueOcean, only stages, but all stages, should be visible with Green balls. branch names (SlaveX) should be a kind of label above the arrow but not a green ball by itself. Therefore, if a branch has no label, then you would see just a line (with its label). If a branch has a serie of N stages, then you would see those stages in that line (with its label), as Alan Wu proposed, I guess. Furthermore, if a stage within a branch consists of a new parallel block or a serie of parallel blocks, then all these sub-stages/branches should be seen as well in their relative branch in the same way. I know there is a question regarding how deep to display such a kind of nested graph, maybe could you add a 'depth' parameter in the page (default: 2?) and display the graph accordingly? Thanks Regards Patrick

Hide

Permalink

Alan Wu added a comment - 2016-12-18 19:53

Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.

Show

Alan Wu added a comment - 2016-12-18 19:53 Regarding how deep the nested graph should show, I think making each of the dot displaying stage collapsible will be quite helpful. Clicking on a parent stage will open up the next level of children stages.

Hide

Permalink

James Dumay added a comment - 2016-12-18 22:19

Alan Wu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.

Show

James Dumay added a comment - 2016-12-18 22:19 Alan Wu we may not allow for infinite nesting in the way that you have described (comes with a lot of design challenges). It's more likely that declarative pipeline will be updated to support a single "sub-stage" under a top level stage.

Hide

Permalink

Alan Wu added a comment - 2016-12-18 23:52

James Dumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages.

I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.

Show

Alan Wu added a comment - 2016-12-18 23:52 James Dumay Sure, I can imagine the difficulties of dealing with layers upon layers of stages. I also agree with you that a specialised syntax to allow multiple parallel brenches to be executed in the same manner without having to copy and paste blocks of code will be very useful.

Hide

Permalink

James Dumay added a comment - 2016-12-19 00:00 - edited

Alan Wu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait

Show

James Dumay added a comment - 2016-12-19 00:00 - edited Alan Wu thats great. FWIW, we want to build the best tool for the job rather than a bunch of round shape things and then fitting them to square holes. Just takes a but longer but I guarantee will be worth the wait

Hide

Permalink

James Dumay added a comment - 2016-12-19 00:04 - edited

Patrick Redivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.

Show

James Dumay added a comment - 2016-12-19 00:04 - edited Patrick Redivo unfortunately it isn't that simple. The visualization isn't a generic graph rendering that can just render things like the way you described without serious changes. Without making any promises a nesting feature is on our radar after the release of Blue Ocean 1.0 early next year.

James Dumay made changes - 2016-12-19 12:00

Link

This issue relates to ~~JENKINS-39464~~ [ ~~JENKINS-39464~~ ]

James Dumay made changes - 2016-12-28 02:41

Link

This issue relates to ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-04 01:52

Link

This issue relates to ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-04 01:53

Link

This issue is duplicated by ~~JENKINS-40619~~ [ ~~JENKINS-40619~~ ]

James Dumay made changes - 2017-01-06 05:54

Rank

Ranked higher

James Dumay made changes - 2017-01-06 05:54

Rank

Ranked lower

James Dumay made changes - 2017-01-09 21:45

Remote Link

This issue links to "Jenkins/Jenkins tried to use a matrix style parallel (Web Link)" [ 15226 ]

Hide

Permalink

James Nord added a comment - 2017-01-13 19:01

I thought this was the reason that stage was made into a block step?

Show

James Nord added a comment - 2017-01-13 19:01 I thought this was the reason that stage was made into a block step?

Hide

Permalink

James Dumay added a comment - 2017-01-14 02:05

James Nord yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good.

Show

James Dumay added a comment - 2017-01-14 02:05 James Nord yes but we failed to figure out how it should be visualized and if it can be visualized. There seems to be different ways users expect the visualization to be displayed and the clarity of the visualization decreases the deeper the stages are nested. As described in the description of this ticket we need to solve both of these issues before we can make it work. I am sure we could implement something but without rigor applied I am not certain it will be good .

Hide

Permalink

Kuber Kaul added a comment - 2017-01-16 22:50

Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

Show

Kuber Kaul added a comment - 2017-01-16 22:50 Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

Hide

Permalink

James Dumay added a comment - 2017-01-16 22:56

Kuber Kaul as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

Show

James Dumay added a comment - 2017-01-16 22:56 Kuber Kaul as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

James Dumay made changes - 2017-01-16 22:57

Description

{panel:title=Support for nested stages is currently unplanned|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Hide

Permalink

Kuber Kaul added a comment - 2017-01-16 23:00

got it, thanks.

Show

Kuber Kaul added a comment - 2017-01-16 23:00 got it, thanks.

Michael Neale made changes - 2017-01-19 01:20

Link

This issue blocks ~~JENKINS-35836~~ [ ~~JENKINS-35836~~ ]

Ian Leitch made changes - 2017-01-20 08:00

Attachment

PipelineGraphProposal.png [ 35539 ]

Hide

Permalink

Ian Leitch added a comment - 2017-01-20 08:00 - edited

Proposal

This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth.

Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview.

1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept)

Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level
In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes.

2. 'stage' and 'parallel_stage' can be nested arbitrarily

The graph only shows the 1st and 2nd levels
If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design)
If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design)
Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage).
Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level
Neither can be called within a 'node' block
Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'?

3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated

stage("Lint") { node { sh "lint" }  }

stage("Build") {
  parallel_stage("Shared") { node { sh "build" } }
  parallel_stage("App 1") { node { sh "build" } }
  parallel_stage("App 2") { node { sh "build" } }
}

stage("Test") {
  parallel_stage("Shared") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
  }

  parallel_stage("App 1") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
  }

  parallel_stage("App 2") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
    parallel_stage("Functional (iPad Air 2 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPad Pro 10.2)") { node { sh "test" } }
  }
}

stage("Collect") { node { sh "collect" }  }

Show

Ian Leitch added a comment - 2017-01-20 08:00 - edited Proposal This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth. Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview. 1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept) Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes. 2. 'stage' and 'parallel_stage' can be nested arbitrarily The graph only shows the 1st and 2nd levels If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design) If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design) Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage). Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level Neither can be called within a 'node' block Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'? 3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated stage( "Lint" ) { node { sh "lint" } } stage( "Build" ) { parallel_stage( "Shared" ) { node { sh "build" } } parallel_stage( "App 1" ) { node { sh "build" } } parallel_stage( "App 2" ) { node { sh "build" } } } stage( "Test" ) { parallel_stage( "Shared" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } } parallel_stage( "App 1" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } } parallel_stage( "App 2" ) { parallel_stage( "Unit (9.3)" ) { node { sh "test" } } parallel_stage( "Unit (10.2)" ) { node { sh "test" } } parallel_stage( "Snapshot (iPhone 7 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPhone 7 10.2)" ) { parallel_stage( "Batch 1" ) { node { sh "test" } } parallel_stage( "Batch 2" { node { sh "test" } } parallel_stage( "Batch 3" ) { node { sh "test" } } } parallel_stage( "Functional (iPad Air 2 10.2)" ) { node { sh "test" } } parallel_stage( "Functional (iPad Pro 10.2)" ) { node { sh "test" } } } } stage( "Collect" ) { node { sh "collect" } }

Hide

Permalink

Alan Wu added a comment - 2017-01-22 22:38

Ian Leitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal?

stage('build and test') {
parallel('Slave1' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
parallel('Slave2' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
}

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

Show

Alan Wu added a comment - 2017-01-22 22:38 Ian Leitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal? stage('build and test') { parallel('Slave1' : { stage('configure') { } stage('build') { } stage('test') { } } parallel('Slave2' : { stage('configure') { } stage('build') { } stage('test') { } } } If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

Hide

Permalink

James Dumay added a comment - 2017-01-22 23:08

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

Alan Wu yes, that is correct - we can't display those right now.

Show

James Dumay added a comment - 2017-01-22 23:08 If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired? Alan Wu yes, that is correct - we can't display those right now.

Hide

Permalink

Ian Leitch added a comment - 2017-01-22 23:53

Alan Wu James Dumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

Show

Ian Leitch added a comment - 2017-01-22 23:53 Alan Wu James Dumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

Hide

Permalink

Alan Wu added a comment - 2017-01-22 23:58

Ian Leitch That's great, it will be very useful to have named_step. Thank you.

Show

Alan Wu added a comment - 2017-01-22 23:58 Ian Leitch That's great, it will be very useful to have named_step. Thank you.

James Dumay made changes - 2017-02-08 21:21

Link

This issue is duplicated by ~~JENKINS-41781~~ [ ~~JENKINS-41781~~ ]

Ryan Hutchison made changes - 2017-02-15 18:05

Link

This issue is related to ~~JENKINS-41156~~ [ ~~JENKINS-41156~~ ]

Hide

Permalink

Jesse Glick added a comment - 2017-02-15 18:47

FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

Show

Jesse Glick added a comment - 2017-02-15 18:47 FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

Hide

Permalink

Michael Neale added a comment - 2017-02-15 23:24

Jesse Glick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

Show

Michael Neale added a comment - 2017-02-15 23:24 Jesse Glick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

James Dumay made changes - 2017-02-15 23:54

Link

This issue blocks ~~JENKINS-42062~~ [ ~~JENKINS-42062~~ ]

Michael Neale made changes - 2017-02-23 04:05

Link

This issue relates to ~~JENKINS-41205~~ [ ~~JENKINS-41205~~ ]

Hide

Permalink

Michael Neale added a comment - 2017-03-20 21:21

There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by:

One level of stages shown in parallel streams of execution
some first class support for matrix of combinations (in paralllel)

I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution).

Show

Michael Neale added a comment - 2017-03-20 21:21 There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by: One level of stages shown in parallel streams of execution some first class support for matrix of combinations (in paralllel) I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution).

Ben Dean made changes - 2017-03-21 14:43

Attachment

example.png [ 36599 ]

Ben Dean made changes - 2017-03-21 18:07

Attachment

pipeline_mockup.png [ 36602 ]

Hide

Permalink

Ben Dean added a comment - 2017-03-21 18:07 - edited

Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages:

Clone SCM
Build
Build documentation on Windows slave (slow)
Custom static analysis tool (slow)
Multiple Test suites
Archive build artifacts and test results

Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO):

Jenkinsfile

stage('Clone') {
    checkout scm
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

parallel([
    buildAndTest: {
        stage('Build') {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build-artifacts'
            }
        }
        stage('Test') {
            testSteps = [:]
            for (suite in testSuites) {
                testSteps[suite] = {
                    node('linux') {
                        unstash 'build-artifacts'
                        sh "run_tests $suite"
                        stash includes: 'test_results/**/*.xml', name: "test-$suite"
                    }
                }
            }
            parallel(testSteps)
        }
    },
    docs: {
        stage('Documentation') {
            node('windows') {
                bat 'build_docs.exe'
            }
        }
    },
    staticAnalysis: {
        stage('Static Analysis') {
            node('linux') {
                sh 'run_static_analysis'
            }
        }
    }
])

stage('Archive') {
    unstash 'build-artifacts'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages.

As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render:

example.dot

digraph {
  graph [rankdir=LR]

  Clone -> Build
  Clone -> Documentation
  Clone -> "Static Analysis"

  Documentation -> Archive
  "Static Analysis" -> Archive

  Build -> "Test suite1"
  Build -> "Test suite2"
  Build -> "Test suite3"

  "Test suite1" -> Archive
  "Test suite2" -> Archive
  "Test suite3" -> Archive
}

Here's what I was thinking it might look like in Blue Ocean

The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this:

Jenkinsfile

stage('Clone') {
    checkout scm
}

stage('Build') {
    parallel([
        build: {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build'
            }
        },
        docs: {
            build job: 'docs', wait: false
        },
        staticAnalysis: {
            build job: 'staticAnalysis', wait: false
        }
    ])
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

stage('Test') {
    testSteps = [:]
    for (suite in testSuites) {
        testSteps[suite] = {
            node('linux') {
                unstash 'build'
                sh "run_tests $suite"
                stash includes: 'test_results/**/*.xml', name: "test-$suite"
            }
        }
    }
    parallel(testSteps)
}

// method to lookup the docs and staticAnalysis jobs we started and see if they're done
@NonCPS
def findSomeBuild(jobName)
    // do Jenkins groovy stuff
    return correctBuildNumber || null
end

stage('Archive') {
    unstash 'build'

    def docsBuildNumber = null    
    waitUntil {
        docsBuildNumber = findSomeBuild('docs')
    }
    
    def staticAnalysisBuildNumber = null
    waitUntil {
        staticAnalysisBuildNumber = findSomeBuild('staticAnalysis')
    }
    
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'docs', selector: [$class: 'SpecificBuildSelector', buildNumber: docsBuildNumber]])
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'staticAnalysis', selector: [$class: 'SpecificBuildSelector', buildNumber: staticAnalysisBuildNumber]])

    unstash 'documentation'
    unstash 'static_analysis'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

Show

Ben Dean added a comment - 2017-03-21 18:07 - edited Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages: Clone SCM Build Build documentation on Windows slave (slow) Custom static analysis tool (slow) Multiple Test suites Archive build artifacts and test results Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO): Jenkinsfile stage( 'Clone' ) { checkout scm } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] parallel([ buildAndTest: { stage( 'Build' ) { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build-artifacts' } } stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build-artifacts' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } }, docs: { stage( 'Documentation' ) { node( 'windows' ) { bat 'build_docs.exe' } } }, staticAnalysis: { stage( 'Static Analysis' ) { node( 'linux' ) { sh 'run_static_analysis' } } } ]) stage( 'Archive' ) { unstash 'build-artifacts' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' } I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages. As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render: example.dot digraph { graph [rankdir=LR] Clone -> Build Clone -> Documentation Clone -> "Static Analysis" Documentation -> Archive "Static Analysis" -> Archive Build -> "Test suite1" Build -> "Test suite2" Build -> "Test suite3" "Test suite1" -> Archive "Test suite2" -> Archive "Test suite3" -> Archive } Here's what I was thinking it might look like in Blue Ocean The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this: Jenkinsfile stage( 'Clone' ) { checkout scm } stage( 'Build' ) { parallel([ build: { node( 'linux' ) { sh 'make' stash includes: 'pkg /**/ *' , name: 'build' } }, docs: { build job: 'docs' , wait: false }, staticAnalysis: { build job: 'staticAnalysis' , wait: false } ]) } testSuites = [ 'suite1' , 'suite2' , 'suite3' ] stage( 'Test' ) { testSteps = [:] for (suite in testSuites) { testSteps[suite] = { node( 'linux' ) { unstash 'build' sh "run_tests $suite" stash includes: 'test_results /**/ *.xml' , name: "test-$suite" } } } parallel(testSteps) } // method to lookup the docs and staticAnalysis jobs we started and see if they're done @NonCPS def findSomeBuild(jobName) // do Jenkins groovy stuff return correctBuildNumber || null end stage( 'Archive' ) { unstash 'build' def docsBuildNumber = null waitUntil { docsBuildNumber = findSomeBuild( 'docs' ) } def staticAnalysisBuildNumber = null waitUntil { staticAnalysisBuildNumber = findSomeBuild( 'staticAnalysis' ) } step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'docs' , selector: [$class: 'SpecificBuildSelector' , buildNumber: docsBuildNumber]]) step([$class: 'CopyArtifact' , filter: 'pkg /**/ *' , projectName: 'staticAnalysis' , selector: [$class: 'SpecificBuildSelector' , buildNumber: staticAnalysisBuildNumber]]) unstash 'documentation' unstash 'static_analysis' for (suite in testSuites) { unstash "test-$suite" } archive 'pkg /**/ *' junit 'test_results /**/ *.xml' }

Hide

Permalink

Laurent Gillet added a comment - 2017-03-21 18:53 - edited

@Ben Dean: I totally agree with your proposition!

We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

Show

Laurent Gillet added a comment - 2017-03-21 18:53 - edited @ Ben Dean : I totally agree with your proposition! We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

Hide

Permalink

Andrew Miller added a comment - 2017-03-28 00:15

May I suggest the following as a means to represent nested stages visually?

Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts.

This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic.

I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

Show

Andrew Miller added a comment - 2017-03-28 00:15 May I suggest the following as a means to represent nested stages visually? Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts. This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic. I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

Hide

Permalink

James Dumay added a comment - 2017-03-28 01:34

Andrew Miller could you draw a picture and upload it here please?

Show

James Dumay added a comment - 2017-03-28 01:34 Andrew Miller could you draw a picture and upload it here please?

Andrew Miller made changes - 2017-04-12 07:16

Attachment

image-2017-04-12-16-16-17-144.png [ 37113 ]

Andrew Miller made changes - 2017-04-12 07:17

Attachment

image-2017-04-12-16-17-38-285.png [ 37114 ]

Hide

Permalink

Andrew Miller added a comment - 2017-04-12 07:19

Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind.

So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final).

When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment):

This process could basically allow for an infinite expansion inwards.

The red/green status would be determined by the final result of all children stages.

Show

Andrew Miller added a comment - 2017-04-12 07:19 Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind. So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final). When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment): This process could basically allow for an infinite expansion inwards. The red/green status would be determined by the final result of all children stages.

Hide

Permalink

Prateek VERMA added a comment - 2017-04-12 08:35

Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise.

In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps.

Lable foo
---O--O--O-----------

s1 s2 s3

Lable bar

---O--O--O-
s1 s2 s3

Show

Prateek VERMA added a comment - 2017-04-12 08:35 Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise. In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps. Lable foo --- O -- O -- O ----------- s1 s2 s3 Lable bar --- O -- O -- O - s1 s2 s3

Prateek VERMA made changes - 2017-04-12 08:35

Assignee

Jesse Glick [ jglick ]

Michael Neale made changes - 2017-04-12 08:37

Assignee

Jesse Glick [ jglick ]

Hide

Permalink

Prateek VERMA added a comment - 2017-04-12 08:39

Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

Show

Prateek VERMA added a comment - 2017-04-12 08:39 Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

Prateek VERMA made changes - 2017-04-12 08:53

Attachment

Proposed Parallel Stages.JPG [ 37115 ]

Hide

Permalink

Prateek VERMA added a comment - 2017-04-12 08:53

Please refer to "Proposed Parallel Stages.JPG" attached here.

Show

Prateek VERMA added a comment - 2017-04-12 08:53 Please refer to "Proposed Parallel Stages.JPG" attached here.

ben powers made changes - 2017-04-12 18:59

Attachment

alternate zoom in.png [ 37120 ]

Hide

Permalink

ben powers added a comment - 2017-04-12 19:06

I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources

As he mentioned, this should provide a UI framework to support additional nesting.

Here's a sketch that might help Andrew feel better about his artwork Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

Show

ben powers added a comment - 2017-04-12 19:06 I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources As he mentioned, this should provide a UI framework to support additional nesting. Here's a sketch that might help Andrew feel better about his artwork Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

Tushar Mahale made changes - 2017-04-14 01:26

Assignee

Tushar Mahale [ tusharmahale ]

James Dumay made changes - 2017-04-14 01:32

Assignee

Tushar Mahale [ tusharmahale ]

Hide

Permalink

Jean Mertz added a comment - 2017-04-27 15:31

I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations.

It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

Show

Jean Mertz added a comment - 2017-04-27 15:31 I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations. It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

James Dumay made changes - 2017-05-01 04:30

Epic Link

JENKINS-35750 [ 171713 ]

JENKINS-43953 [ 181485 ]

James Dumay made changes - 2017-06-19 01:56

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2017-06-19 01:56

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

James Dumay made changes - 2017-06-19 01:57

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=#3878de|titleColor=white}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jean-Paul G made changes - 2017-06-22 04:20

Link

This issue is related to ~~JENKINS-44820~~ [ ~~JENKINS-44820~~ ]

James Dumay made changes - 2017-08-13 05:48

Link

This issue is duplicated by ~~JENKINS-45664~~ [ ~~JENKINS-45664~~ ]

Joerg Schwaerzler made changes - 2017-08-14 08:29

Attachment

image-2017-08-14-10-29-55-373.png [ 39258 ]

Joerg Schwaerzler made changes - 2017-08-14 08:34

Attachment

image-2017-08-14-10-34-10-452.png [ 39259 ]

Hide

Permalink

Joerg Schwaerzler added a comment - 2017-08-14 08:36 - edited

How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.:

stage ('Build') {
    // Build something
}
parallel('Test1' : {
    stage('JUnit') {
    }
    stage('Firefox') {
    }
}, 'Test2' : {
    stage('DBUnit') {
    }
    stage('Edge') {
    }
}, 'Test3' : {
    stage('Jasmine') {
    }
    stage('Safari') {
    }    
}
stage ('Dev') {
}

Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

Show

Joerg Schwaerzler added a comment - 2017-08-14 08:36 - edited How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.: stage ( 'Build' ) { // Build something } parallel( 'Test1' : { stage( 'JUnit' ) { } stage( 'Firefox' ) { } }, 'Test2' : { stage( 'DBUnit' ) { } stage( 'Edge' ) { } }, 'Test3' : { stage( 'Jasmine' ) { } stage( 'Safari' ) { } } stage ( 'Dev' ) { } Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

Hide

Permalink

Michael Neale added a comment - 2017-08-14 08:37

Joerg Schwaerzler that I think is what most people agree on as some kind of next step for this.

Show

Michael Neale added a comment - 2017-08-14 08:37 Joerg Schwaerzler that I think is what most people agree on as some kind of next step for this.

Hide

Permalink

Joerg Schwaerzler added a comment - 2017-08-14 08:40

Great than, was not sure about that.

Show

Joerg Schwaerzler added a comment - 2017-08-14 08:40 Great than, was not sure about that.

Joerg Schwaerzler made changes - 2017-08-14 08:41

Attachment

image-2017-08-14-10-29-55-373.png [ 39258 ]

Hide

Permalink

Florian Manschwetus added a comment - 2017-08-14 08:43 - edited

Joerg Schwaerzler +1, no more words on this needed from my side.

Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

Show

Florian Manschwetus added a comment - 2017-08-14 08:43 - edited Joerg Schwaerzler +1, no more words on this needed from my side. Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

Hide

Permalink

Joerg Schwaerzler added a comment - 2017-08-14 09:00

Florian Manschwetus

Sorry for that - just wanted to be crystal clear about what I think should be the next step.

Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts.

But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks

However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

Show

Joerg Schwaerzler added a comment - 2017-08-14 09:00 Florian Manschwetus Sorry for that - just wanted to be crystal clear about what I think should be the next step. Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts. But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

James Dumay made changes - 2017-08-28 17:39

Link

This issue is duplicated by ~~JENKINS-46345~~ [ ~~JENKINS-46345~~ ]

Michael Neale made changes - 2017-10-12 01:59

Remote Link

This issue links to "Page (Jenkins Wiki)" [ 17861 ]

Henri Anttila made changes - 2017-10-19 05:22

Attachment

screenshot-1.png [ 40062 ]

Henri Anttila made changes - 2017-10-19 05:24

Attachment

image-2017-10-19-08-24-00-504.png [ 40063 ]

Hide

Permalink

Henri Anttila added a comment - 2017-10-19 05:25

Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so:

These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

Show

Henri Anttila added a comment - 2017-10-19 05:25 Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so: These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

Hide

Permalink

Michael Neale added a comment - 2017-10-19 05:27

Henri Anttila do you mean more like a matrix of combinations?

Show

Michael Neale added a comment - 2017-10-19 05:27 Henri Anttila do you mean more like a matrix of combinations?

Hide

Permalink

Josh McDonald added a comment - 2017-10-19 07:22

Maybe we should start thinking about a BO-style renderer for matrix jobs?

Show

Josh McDonald added a comment - 2017-10-19 07:22 Maybe we should start thinking about a BO-style renderer for matrix jobs?

Hide

Permalink

Michael Neale added a comment - 2017-10-19 07:27

Josh McDonald yes - and I know Andrew Bayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed.

Show

Michael Neale added a comment - 2017-10-19 07:27 Josh McDonald yes - and I know Andrew Bayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed.

Hide

Permalink

Henri Anttila added a comment - 2017-10-19 09:30

I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization.

For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary.

After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

Show

Henri Anttila added a comment - 2017-10-19 09:30 I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization. For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary. After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

Hide

Permalink

Chris Majoros added a comment - 2017-10-19 17:24

I am actually looking to do the same thing as Henri Anttila

Show

Chris Majoros added a comment - 2017-10-19 17:24 I am actually looking to do the same thing as Henri Anttila

Hide

Permalink

Mathew Joseph added a comment - 2017-11-19 20:43

Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

Show

Mathew Joseph added a comment - 2017-11-19 20:43 Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

Hide

Permalink

Michael Neale added a comment - 2017-11-19 22:44

Mathew Joseph not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet.

Show

Michael Neale added a comment - 2017-11-19 22:44 Mathew Joseph not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet.

James Dumay made changes - 2017-11-30 23:23

Remote Link

This issue links to "CloudBees Internal UX-584 (Web Link)" [ 18213 ]

Hide

Permalink

Nicholas Roth added a comment - 2018-01-02 19:18

This would be really nice to have!

Show

Nicholas Roth added a comment - 2018-01-02 19:18 This would be really nice to have!

Hide

Permalink

Roy Arnon added a comment - 2018-01-03 15:04 - edited

+1 from me as well.

In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules. We are currently converting them to pipelines, and we parallelize a lot of steps:

Build Jars
Package & Test (Parallel)
1. Build RPM
2. Build Docker
3. Run Tests (Parallel)
Promote build

We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps.

Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps.

I also think the Andrew Miller suggestion seems like the best option.

Show

Roy Arnon added a comment - 2018-01-03 15:04 - edited +1 from me as well. In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules. We are currently converting them to pipelines, and we parallelize a lot of steps: Build Jars Package & Test (Parallel) Build RPM Build Docker Run Tests (Parallel) Promote build We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps. Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps. I also think the Andrew Miller suggestion seems like the best option.

Hide

Permalink

Florian Wilhelm added a comment - 2018-01-08 16:12

I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved.

I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future?

Best regards,
Florian

Show

Florian Wilhelm added a comment - 2018-01-08 16:12 I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved. I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future? Best regards, Florian

Gabriel Loewen made changes - 2018-03-09 16:10

Attachment

Before.PNG [ 41786 ]

Gabriel Loewen made changes - 2018-03-09 16:10

Attachment

After.png [ 41787 ]

Hide

Permalink

Gabriel Loewen added a comment - 2018-03-09 16:14

Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following:

pipeline {
    agent any
    stages {
        stage("Testing") {
            steps {
                script {
                    stage("A") {
                        parallel "1": {
                            println "1"
			    sleep 500
                        }, "2": {
                            println "2"
			    sleep 500
                        }, "3": {
                            println "3"
			    sleep 500
                        }
                    }
                }
            }
        }
    }
}

Then while this pipeline is running I see the following in Blue Ocean:

But after the pipeline finishes the nested stage and the parent stage get merged:

What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

Show

Gabriel Loewen added a comment - 2018-03-09 16:14 Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following: pipeline { agent any stages { stage( "Testing" ) { steps { script { stage( "A" ) { parallel "1" : { println "1" sleep 500 }, "2" : { println "2" sleep 500 }, "3" : { println "3" sleep 500 } } } } } } } Then while this pipeline is running I see the following in Blue Ocean: But after the pipeline finishes the nested stage and the parent stage get merged: What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

Hide

Permalink

Costin Caraivan added a comment - 2018-03-15 12:06 - edited

This is quite important for many CI scenarios. Quick example:

first stage: build - executed on a build node
parallel launch of 3 logical "blocks":
___ 1 block for 1 type of integration test - executed on a specialized integration node

___ 1 block for 1 other type of integration test - executed on another specialized integration node
___ 1 bigger block for deployment and functional tests:
___1 stage for deployment - executed on the target app environment
___1 stage for functional tests - executed on a specialized functional test runner node

Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless.

In my opinion perfect is the enemy of good.

Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough.

This should simplify the implementation of both the actual pipeline code and also the visualization.

Show

Costin Caraivan added a comment - 2018-03-15 12:06 - edited This is quite important for many CI scenarios. Quick example: first stage: build - executed on a build node parallel launch of 3 logical "blocks": ___ 1 block for 1 type of integration test - executed on a specialized integration node ___ 1 block for 1 other type of integration test - executed on another specialized integration node ___ 1 bigger block for deployment and functional tests: ___1 stage for deployment - executed on the target app environment ___1 stage for functional tests - executed on a specialized functional test runner node Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless. In my opinion perfect is the enemy of good. Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough. This should simplify the implementation of both the actual pipeline code and also the visualization.

Hide

Permalink

Anthony BOUQUET added a comment - 2018-03-20 11:00

+1 for Costin Caraivan

Have long running step and can't execute it in a parallel way because of this nested pipeline limitation.

3 level of nested pipeline should be enought

Show

Anthony BOUQUET added a comment - 2018-03-20 11:00 +1 for Costin Caraivan Have long running step and can't execute it in a parallel way because of this nested pipeline limitation. 3 level of nested pipeline should be enought

Hide

Permalink

Hokwang Lee added a comment - 2018-03-28 13:57 - edited

+1

I think this is really really needed for making flexible pipeline.

Because of feature toggling,

we need

checkout - build - test

... (so many)

(only different is source code checkouted and build command line)

If this feature can not be implemented, I will search another tool.

Show

Hokwang Lee added a comment - 2018-03-28 13:57 - edited +1 I think this is really really needed for making flexible pipeline. Because of feature toggling, we need checkout - build - test checkout - build - test checkout - build - test ... (so many) (only different is source code checkouted and build command line) If this feature can not be implemented, I will search another tool.

Hide

Permalink

Todd Densmore added a comment - 2018-06-20 21:36

Intel is also very interested in this visualization enhancement.

Show

Todd Densmore added a comment - 2018-06-20 21:36 Intel is also very interested in this visualization enhancement.

Hide

Permalink

Jenn Briden added a comment - 2018-06-20 23:03

We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227.

Show

Jenn Briden added a comment - 2018-06-20 23:03 We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227.

Jenn Briden made changes - 2018-06-20 23:49

Assignee

CloudBees Inc. [ cloudbees ]

Hide

Permalink

Joerg Schwaerzler added a comment - 2018-06-21 06:09

Great to hear

Two questions:

Are you also planning to support the scripted pipeline?
As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.

Show

Joerg Schwaerzler added a comment - 2018-06-21 06:09 Great to hear Two questions: Are you also planning to support the scripted pipeline? As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.

Jenn Briden made changes - 2018-07-06 21:15

Status

Open [ 1 ]

In Progress [ 3 ]

Jenn Briden made changes - 2018-07-06 21:18

Description

{panel:title=Support for nested stages is currently unplanned and is not actively being worked on|titleBGColor=lightgrey}
There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}

*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}

*Example 2*

{code}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}

There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}
*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jenn Briden made changes - 2018-07-06 21:19

Description

{panel}

There is some debate on how this concept can be visualised in Blue Ocean. Its possible to have infinitely nested stages that would be impossible for the visualization to display correctly.

It's also impossible for us to correctly decide if a inner stage should be used as a "label" for a set of steps or if it is an actual stage. Labelling using {{stage}} needs to be revisited in Pipeline Script.

The Blue Ocean team needs to refine the design for the pipeline graph for this to be supported.
{panel}
*Original ticket*
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}
In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Jenn Briden made changes - 2018-07-06 21:20

Summary

Support nested stages in the Pipeline Graph

View sequential stages in the pipeline visualization graph

Hide

Permalink

Domagoj Smoljanovic added a comment - 2018-08-01 09:04

I see that the above PR is merged but issue is still in progress. Is this still being worked on?

When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that.

Thanx

Show

Domagoj Smoljanovic added a comment - 2018-08-01 09:04 I see that the above PR is merged but issue is still in progress. Is this still being worked on? When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that. Thanx

Hide

Permalink

Josh McDonald added a comment - 2018-08-01 09:14

AFAIK there's no plan to support nested parallels at the moment.

Show

Josh McDonald added a comment - 2018-08-01 09:14 AFAIK there's no plan to support nested parallels at the moment.

Hide

Permalink

Daniel Kurzynski added a comment - 2018-08-01 09:21

We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

Show

Daniel Kurzynski added a comment - 2018-08-01 09:21 We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

Hide

Permalink

Mathias Hasselmann added a comment - 2018-08-01 10:24

I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

Show

Mathias Hasselmann added a comment - 2018-08-01 10:24 I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

Gert van Dijk made changes - 2018-08-14 08:27

Attachment

jenkins-weird-masked2.png [ 43737 ]

Hide

Permalink

Gert van Dijk added a comment - 2018-08-14 08:33 - edited

First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway.

However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent.

It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage. In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages.

Sub-stages are shown during the run, though.

As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues.
(Seeing this on 1.7.2 / Core 2.121.2)

Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

Show

Gert van Dijk added a comment - 2018-08-14 08:33 - edited First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway. However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent. It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage . In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages. Sub-stages are shown during the run , though. As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues. (Seeing this on 1.7.2 / Core 2.121.2) Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

Hide

Permalink

Devin Nusbaum added a comment - 2018-08-17 19:13 - edited

Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of ~~JENKINS-49050~~ (although note that I believe that issue is specific to declarative pipeline).

Show

Devin Nusbaum added a comment - 2018-08-17 19:13 - edited Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of JENKINS-49050 (although note that I believe that issue is specific to declarative pipeline).

Devin Nusbaum made changes - 2018-08-17 19:13

Link

This issue duplicates ~~JENKINS-49050~~ [ ~~JENKINS-49050~~ ]

Devin Nusbaum made changes - 2018-08-17 19:13

Resolution		Duplicate [ 3 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Devin Nusbaum made changes - 2018-08-17 19:14

Released As

Blue Ocean 1.8.0

Hide

Permalink

Joerg Schwaerzler added a comment - 2018-08-18 05:37

Devin Nusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?

Show

Joerg Schwaerzler added a comment - 2018-08-18 05:37 Devin Nusbaum So for nested stages in scripted pipelines, do we need to open another feature request - or is there already something in place?

Hide

Permalink

Devin Nusbaum added a comment - 2018-08-20 13:12

Joerg Schwaerzler I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

Show

Devin Nusbaum added a comment - 2018-08-20 13:12 Joerg Schwaerzler I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

Hide

Permalink

Daniel Kurzynski added a comment - 2018-08-20 13:20 - edited

I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization.
Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

Show

Daniel Kurzynski added a comment - 2018-08-20 13:20 - edited I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization. Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

Hide

Permalink

Devin Nusbaum added a comment - 2018-08-20 13:24

Daniel Kurzynski Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

Show

Devin Nusbaum added a comment - 2018-08-20 13:24 Daniel Kurzynski Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

Hide

Permalink

Joerg Schwaerzler added a comment - 2018-08-21 12:28

Devin Nusbaum: Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

Show

Joerg Schwaerzler added a comment - 2018-08-21 12:28 Devin Nusbaum : Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

Hide

Permalink

Joerg Schwaerzler added a comment - 2018-08-21 12:41

For those interested: I raised a new fr here: ~~JENKINS-53162~~

Show

Joerg Schwaerzler added a comment - 2018-08-21 12:41 For those interested: I raised a new fr here: JENKINS-53162

Devin Nusbaum made changes - 2018-08-21 13:36

Link

This issue relates to ~~JENKINS-53162~~ [ ~~JENKINS-53162~~ ]

Hide

Permalink

Phil Segal added a comment - 2018-10-10 09:35

Josh McDonald is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN . Something like pipeline_mockups.png from above would be amazing

Show

Phil Segal added a comment - 2018-10-10 09:35 Josh McDonald is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN . Something like pipeline_mockups.png from above would be amazing

Hide

Permalink

Daniel Kurzynski added a comment - 2018-10-11 08:52

Devin Nusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

Show

Daniel Kurzynski added a comment - 2018-10-11 08:52 Devin Nusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

Jesse Glick made changes - 2018-10-11 14:04

Link

This issue relates to JENKINS-54010 [ JENKINS-54010 ]

zengqiao liu made changes - 2018-12-17 20:59

Description

{panel}
In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

{panel}

In progress - one level of visualization below parent stage. Additional research and feedback will help us determine future visualizations.
{panel}
I've tried using nested stages in order to group each shell script alone to have a meaningful data, however this broken the UI of blueocean and it stopped it from showing showing the variant steps.

*Example 1*
{code:java}
def test = [:]

test["a"] = {
    stage ("a") {
        stage ("ab") {
            sh "echo stage abc"
        }
        stage ("xyz") {
            sh "echo stage xyz"
        }
    }
}

test["b"] = {
    stage ("b") {
        stage ("bb") {
            sh "echo stage bb"
        }
        stage ("bxz") {
            sh "echo stagebxyz"
        }
    }
}
node {
   //stage 'start'
   parallel test
   stage ('middle') {
       sh "echo middle"
   }

}
{code}
*Example 2*
{code:java}
node() {
  stage('Build') {
    println 'I prepare the build for the parallel steps'
  }

  stage('Test') {
   parallel (
"win7-vs2012" : { stage("checkout") { }; stage("build") { }; stage("test") { } },
"win10-vs2015" : { stage("checkout") { }; stage("build") { }; stage("test") { }},
"linux-gcc5" : { stage("checkout") { }; stage("build") { }; stage("test") { } }
)
  }
}
{code}

Hide

Permalink

Ahmed Sghaier added a comment - 2019-01-11 13:02

Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647

Should we create a separate issue for this?

Show

Ahmed Sghaier added a comment - 2019-01-11 13:02 Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647 Should we create a separate issue for this?

Anentropic made changes - 2020-11-26 17:15

Attachment

current.png [ 53474 ]

Anentropic made changes - 2020-11-26 17:16

Attachment

Jenkins-substages.png [ 53475 ]

Hide

Permalink

Anentropic added a comment - 2020-11-26 17:17

This is a very long thread so forgive me if this is already covered...

Can anyone explain why a stage that contains sub-stages is displayed as a preceding node instead of a container?

Take this example:

The stage "Test" performs no actions of its own, but it contains the sub-stages that follow.

Wouldn't it make more sense to display it as a container, something like:

Show

Anentropic added a comment - 2020-11-26 17:17 This is a very long thread so forgive me if this is already covered... Can anyone explain why a stage that contains sub-stages is displayed as a preceding node instead of a container? Take this example: The stage "Test" performs no actions of its own, but it contains the sub-stages that follow. Wouldn't it make more sense to display it as a container, something like:

Jenkins

View sequential stages in the pipeline visualization graph

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates