[JENKINS-39839] Missing StandardChunkVisitor.parallelBranchEnd() and parallelEnd() events if there are nested parallels - Jenkins Jira

Vivek Pandey created issue - 2016-11-17 20:25

Vivek Pandey made changes - 2016-11-17 20:25

Field	Original Value	New Value
Priority	Minor [ 4 ]	Major [ 3 ]

Vivek Pandey made changes - 2016-11-17 20:42

Summary

Missing parallelBranchEnd and parallelEnd events if there are nested parallels

Missing StandardChunkVisitor.parallelBranchEnd() and parallelEnd() events if there are nested parallels

Vivek Pandey made changes - 2016-11-17 20:43

Description

For the given pipeline script, secondLevelNestedBranch1 and secondLevelNestedBranch2 parallel branches are nested inside nestedBranch.

There are missing parallelEnd and parallelBranchEnd event for *nestedBranch*. This causes computing incorrect DAG and status of individual branches.

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         },secondLevelNestedBranch2: {
           echo 'secondLevelNestedBranch2'
         }
       }
     },
    failFast: false
   }
}
{code}

If there are only one parallel branches in nested parallel then no parallelEnd or parallelBranchEnd event is fired. See the script below:

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         }
       }
     },
    failFast: false
   }
}
{code}

For the given pipeline script, secondLevelNestedBranch1 and secondLevelNestedBranch2 parallel branches are nested inside nestedBranch.

There are missing StandardChunkVisitor.parallelEnd() and StandardChunkVisitor.parallelBranchEnd() events for *nestedBranch*. This causes computing incorrect DAG and status of individual branches. Blueocean extends StandardChunkVisitor to process events and create it's DAG.

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         },secondLevelNestedBranch2: {
           echo 'secondLevelNestedBranch2'
         }
       }
     },
    failFast: false
   }
}
{code}

If there are only one parallel branches in nested parallel then no parallelEnd or parallelBranchEnd event is fired. See the script below:

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         }
       }
     },
    failFast: false
   }
}
{code}

James Dumay made changes - 2016-11-17 23:38

Labels

blueocean

Vivek Pandey added a comment - 2016-11-18 05:59

Looks like, firing of parallel events are broken in general. Below script has two stages, each stage with couple parallels, parallelStart event is not called. The pipeline script below has s2.0 and s2.1 are still running so parallelBranchEnd is not fired, understandable, but I expect parallelStart should have been fired.

I collect branches in parallelStart and associate them with relevant stage, reset states etc. so this breaks association of parallel branches with the stage it belongs to. See https://issues.jenkins-ci.org/browse/JENKINS-39847

chunkEnd=> id: 39, name: End of Pipeline, function: End of Pipeline
chunkEnd=> id: 35, name: Stage : Body : End, function: }
  StartNode: org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode[id=24]

parallelEnd=> id: 20, StartNode: 12, name: Execute in parallel : Start, function: parallel
parallelBranchStart=> id: 31, name: Branch: S2.1, function: { (Branch: S2.1)
parallelBranchStart=> id: 26, name: Branch: S2.0, function: { (Branch: S2.0)


handleChunkDone=> id: 24, name: S2, function: { (S2)
chunkStart=> id: 24, name: S2, function: { (S2)


chunkEnd=> id: 21, name: Stage : Body : End, function: }
  StartNode: org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode[id=6]

parallelEnd=> id: 20, StartNode: 12, name: Execute in parallel : Start, function: parallel

parallelBranchEnd=> id: 19, StartNode: 15, name: Branch: S1.1.2, function: { (Branch: S1.1.2)
parallelBranchStart=> id: 15, name: Branch: S1.1.2, function: { (Branch: S1.1.2)

parallelBranchEnd=> id: 18, StartNode: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1)
parallelBranchStart=> id: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1)

parallelStart=> id: 12, name: Execute in parallel : Start, function: parallel
  branch=> id: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1)

parallelBranchStart=> id: 8, name: Branch: S1.0, function: { (Branch: S1.0)
handleChunkDone=> id: 6, name: S1, function: { (S1)
chunkStart=> id: 6, name: S1, function: { (S1)

node {
  stage('S1') {  
    parallel 'S1.0': { sh('sleep 10') }
    
    parallel 'S1.1.1': { sh('sleep 10') }, 'S1.1.2': { sh('sleep 10') }
  }
    stage('S2') {  
    parallel 'S2.0': { sh('sleep 10') }
    
    parallel 'S2.1': { sh('sleep 10') }
  }
}

Vivek Pandey added a comment - 2016-11-18 05:59 Looks like, firing of parallel events are broken in general. Below script has two stages, each stage with couple parallels, parallelStart event is not called. The pipeline script below has s2.0 and s2.1 are still running so parallelBranchEnd is not fired, understandable, but I expect parallelStart should have been fired. I collect branches in parallelStart and associate them with relevant stage, reset states etc. so this breaks association of parallel branches with the stage it belongs to. See https://issues.jenkins-ci.org/browse/JENKINS-39847 chunkEnd=> id: 39, name: End of Pipeline, function: End of Pipeline chunkEnd=> id: 35, name: Stage : Body : End, function: } StartNode: org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode[id=24] parallelEnd=> id: 20, StartNode: 12, name: Execute in parallel : Start, function: parallel parallelBranchStart=> id: 31, name: Branch: S2.1, function: { (Branch: S2.1) parallelBranchStart=> id: 26, name: Branch: S2.0, function: { (Branch: S2.0) handleChunkDone=> id: 24, name: S2, function: { (S2) chunkStart=> id: 24, name: S2, function: { (S2) chunkEnd=> id: 21, name: Stage : Body : End, function: } StartNode: org.jenkinsci.plugins.workflow.cps.nodes.StepStartNode[id=6] parallelEnd=> id: 20, StartNode: 12, name: Execute in parallel : Start, function: parallel parallelBranchEnd=> id: 19, StartNode: 15, name: Branch: S1.1.2, function: { (Branch: S1.1.2) parallelBranchStart=> id: 15, name: Branch: S1.1.2, function: { (Branch: S1.1.2) parallelBranchEnd=> id: 18, StartNode: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1) parallelBranchStart=> id: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1) parallelStart=> id: 12, name: Execute in parallel : Start, function: parallel branch=> id: 14, name: Branch: S1.1.1, function: { (Branch: S1.1.1) parallelBranchStart=> id: 8, name: Branch: S1.0, function: { (Branch: S1.0) handleChunkDone=> id: 6, name: S1, function: { (S1) chunkStart=> id: 6, name: S1, function: { (S1) node { stage( 'S1' ) { parallel 'S1.0' : { sh( 'sleep 10' ) } parallel 'S1.1.1' : { sh( 'sleep 10' ) }, 'S1.1.2' : { sh( 'sleep 10' ) } } stage( 'S2' ) { parallel 'S2.0' : { sh( 'sleep 10' ) } parallel 'S2.1' : { sh( 'sleep 10' ) } } }

Vivek Pandey made changes - 2016-11-18 06:03

Link

This issue blocks JENKINS-39847 [ JENKINS-39847 ]

Vivek Pandey made changes - 2016-11-18 06:05

Description

For the given pipeline script, secondLevelNestedBranch1 and secondLevelNestedBranch2 parallel branches are nested inside nestedBranch.

There are missing StandardChunkVisitor.parallelEnd() and StandardChunkVisitor.parallelBranchEnd() events for *nestedBranch*. This causes computing incorrect DAG and status of individual branches. Blueocean extends StandardChunkVisitor to process events and create it's DAG.

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         },secondLevelNestedBranch2: {
           echo 'secondLevelNestedBranch2'
         }
       }
     },
    failFast: false
   }
}
{code}

If there are only one parallel branches in nested parallel then no parallelEnd or parallelBranchEnd event is fired. See the script below:

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         }
       }
     },
    failFast: false
   }
}
{code}

*workflow-api plugin version 2.6.*

For the given pipeline script, secondLevelNestedBranch1 and secondLevelNestedBranch2 parallel branches are nested inside nestedBranch.

There are missing StandardChunkVisitor.parallelEnd() and StandardChunkVisitor.parallelBranchEnd() events for *nestedBranch*. This causes computing incorrect DAG and status of individual branches. Blueocean extends StandardChunkVisitor to process events and create it's DAG.

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         },secondLevelNestedBranch2: {
           echo 'secondLevelNestedBranch2'
         }
       }
     },
    failFast: false
   }
}
{code}

If there are only one parallel branches in nested parallel then no parallelEnd or parallelBranchEnd event is fired. See the script below:

{code}
node {
   stage ('test') {
     echo ('Testing');
     parallel nestedBranch: {
       echo 'nested Branch'
       stage('nestedBranchStage') {
         echo 'running nestedBranchStage'
         parallel secondLevelNestedBranch1: {
           echo 'secondLevelNestedBranch1'
         }
       }
     },
    failFast: false
   }
}
{code}

Sam Van Oort made changes - 2016-11-18 18:12

Assignee

Sam Van Oort [ svanoort ]

Sam Van Oort added a comment - 2017-02-10 20:36

vivek I probably discussed this in the phone at the time, but for the record: it's impossible to guarantee parallel/branch end events are always fired in the case where there is a parallel with only one branch, because:

For an incomplete branch block (still running) you may not have an end block
If there is not more than one current head node (current execution) you have no way to know it's inside a parallel

This is best handled in your visitor by retaining nodes in case a parallel branch start is discovered (for incomplete flows at least).

I can, however provide guarantees that the parallel starts are correctly handled (and they need to be), and am hardening tests for this case.

Sam Van Oort added a comment - 2017-02-10 20:36 vivek I probably discussed this in the phone at the time, but for the record: it's impossible to guarantee parallel/branch end events are always fired in the case where there is a parallel with only one branch, because: For an incomplete branch block (still running) you may not have an end block If there is not more than one current head node (current execution) you have no way to know it's inside a parallel This is best handled in your visitor by retaining nodes in case a parallel branch start is discovered (for incomplete flows at least). I can, however provide guarantees that the parallel starts are correctly handled (and they need to be), and am hardening tests for this case.

Sam Van Oort made changes - 2017-02-22 15:24

Status

Open [ 1 ]

In Progress [ 3 ]

Sam Van Oort made changes - 2017-02-22 15:24

Status

In Progress [ 3 ]

In Review [ 10005 ]

Sam Van Oort added a comment - 2017-02-22 15:25

vivek This is awaiting your review (along with other fixes) in https://github.com/jenkinsci/workflow-api-plugin/pull/33

Sam Van Oort added a comment - 2017-02-22 15:25 vivek This is awaiting your review (along with other fixes) in https://github.com/jenkinsci/workflow-api-plugin/pull/33

SCM/JIRA link daemon added a comment - 2017-03-02 21:26

Code changed in jenkins
User: Sam Van Oort
Path:
src/main/java/org/jenkinsci/plugins/workflow/graphanalysis/ForkScanner.java
src/test/java/org/jenkinsci/plugins/workflow/graphanalysis/ForkScannerTest.java
http://jenkins-ci.org/commit/workflow-api-plugin/96e2c2ed99384a9c9f4fb80cc8f8876d9b225504
Log:
Fix a couple forkscanner testcase quirks, and put ~~JENKINS-39839~~ to bed along with ~~JENKINS-39841~~

SCM/JIRA link daemon added a comment - 2017-03-02 21:26 Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/graphanalysis/ForkScanner.java src/test/java/org/jenkinsci/plugins/workflow/graphanalysis/ForkScannerTest.java http://jenkins-ci.org/commit/workflow-api-plugin/96e2c2ed99384a9c9f4fb80cc8f8876d9b225504 Log: Fix a couple forkscanner testcase quirks, and put JENKINS-39839 to bed along with JENKINS-39841

Sam Van Oort added a comment - 2017-03-06 18:26

Fixed with explicit test coverage in workflow-api 2.12

Sam Van Oort added a comment - 2017-03-06 18:26 Fixed with explicit test coverage in workflow-api 2.12

Sam Van Oort made changes - 2017-03-06 18:26

Resolution		Fixed [ 1 ]
Status	In Review [ 10005 ]	Closed [ 6 ]

CloudBees Inc. made changes - 2017-12-05 05:39

Remote Link

This issue links to "CloudBees Internal OSS-1691 (Web Link)" [ 18610 ]

Jenkins

Missing StandardChunkVisitor.parallelBranchEnd() and parallelEnd() events if there are nested parallels

Details

Description

Attachments

Issue Links

Activity

People

Dates