Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
-
Declarative backlog
Description
This issue is based on JENKINS-45455 which solves the issue of restarting a Jenkins Pipeline from Top-Level stages.
Description
When building on multiple platforms, the builds are usually ran in parallel. We should be able to replay the build of a single platform. (e.g. if the Win64 build fails, we do not want to retrigger the already successful Macos and Linux builds)
In a multiplatform build pipeline, the steps taking the most time are the "build" steps themselves and they are running in parallel to get fast build times. Restarting from a top level in that context boils down to re-running the whole job.
Use cases
- Restarting a Jenkins Pipeline from a sub-stage or parallel stage
- Given the following graph, we should be able to replay from any stage (e.g. Build-Win64, Test-MacOs, Deploy) :
Build-Win64 -> Test-Win64 -| Build-Win32 -> Test-Win32 -|-> Deploy Build-MacOS -> Test-MacOS -| Build-Linux -> Test-Linux -|
Attachments
Issue Links
- is related to
-
-
- Closed
-
Activity
You can use something like this:
#!groovy def withCheck(String blockName, Closure closure) { script { def buildStage = true catchError(message: 'check previous build status', stageResult:'SUCCESS', buildResult: 'SUCCESS') { unstash name:"${blockName}" buildStage = false } if (buildStage) { closure.call() writeFile file: "${blockName}", text: "1" stash name: "${blockName}", includes: "${blockName}" } } } pipeline { agent none options { preserveStashes() } stages { stage("Build and test") { parallel() { stage("Build/Test Win64") { agent { label 'master' } steps { withCheck("build-deploy-Win64") { echo "test" } withCheck("test-Win64") { echo "test" } } } stage("Build/Test Win32") { agent { label 'master' } steps { withCheck("build-deploy-Win32") { echo "test" } withCheck("test-Win32") { echo "test" } } } } } stage("Deploy") { agent { label 'master' } steps { withCheck("build-deploy-win64") { echo "test" } withCheck("test-win64") { echo "test" } } } } }
Gabriel Herisanu
added a comment - - edited You can use something like this:
#!groovy
def withCheck( String blockName, Closure closure) {
script {
def buildStage = true
catchError(message: 'check previous build status' , stageResult: 'SUCCESS' , buildResult: 'SUCCESS' ) {
unstash name: "${blockName}"
buildStage = false
}
if (buildStage) {
closure.call()
writeFile file: "${blockName}" , text: "1"
stash name: "${blockName}" , includes: "${blockName}"
}
}
}
pipeline {
agent none
options {
preserveStashes()
}
stages {
stage( "Build and test" ) {
parallel() {
stage( "Build/Test Win64" ) {
agent {
label 'master'
}
steps {
withCheck( "build-deploy-Win64" ) {
echo "test"
}
withCheck( "test-Win64" ) {
echo "test"
}
}
}
stage( "Build/Test Win32" ) {
agent {
label 'master'
}
steps {
withCheck( "build-deploy-Win32" ) {
echo "test"
}
withCheck( "test-Win32" ) {
echo "test"
}
}
}
}
}
stage( "Deploy" ) {
agent {
label 'master'
}
steps {
withCheck( "build-deploy-win64" ) {
echo "test"
}
withCheck( "test-win64" ) {
echo "test"
}
}
}
}
}
gabih This bug is about declarative language features - see the Epic Link and Sprint tagging above.
Your suggestion isn't declarative.
gabih That actually looks like a neat trick. Will check whether we will be able to use it in pure scripted pipeline 
gabih Can you confirm that the preserveStashes option will be honored by any stash/unstash called from within the scripted block? Currently it seems like those stashes are not preserved on our Jenkins instance when calling stash from within the script block. But maybe we just need an update of Jenkins or some plugin...
Gabriel Herisanu
added a comment - macdrega preserveStashes(/buildCount: 5/) works if the pipeline is started form the same Jenkins instance
Gabriel Herisanu
added a comment - macdrega preserveStashes(/ buildCount: 5 /) works if the pipeline is started form the same Jenkins instance What I did not realize is that preserverStashes will only work when restarting from some stage... Looks better now - thanks.
Zak Taccardi
added a comment - Is the matrix feature blocked by a lack of support for restarting here?
Zak Taccardi
added a comment - Is the matrix feature blocked by a lack of support for restarting here? 
Mateusz Filipek
added a comment - My version of workaround built on solution from gabih. Because stash can preserve to maximum of 50 builds I've prepared it on Archive artifacts and additionally it doesn't keep a stash in memory. It also requires Copy Artifact plugin - https://plugins.jenkins.io/copyartifact/
def withCheck(String blockName, Closure closure) { script { def buildStage = true catchError(message: 'check previous build status', stageResult:'SUCCESS', buildResult: 'SUCCESS') { copyArtifactsFromRestartedBuild(context.env.JOB_NAME, context.env.BUILD_NUMBER, "stageStates/${blockName}") buildStage = false } if (buildStage) { closure.call() writeFile file: "${blockName}", text: "1" archiveArtifacts artifacts: "stageStates/${blockName}" } } } def getBuildActions(String jobName, int buildNumber) { return Jenkins.instance.getItemByFullName(jobName).getBuildByNumber(buildNumber).properties.actions } def filterBuildAction(def buildActions, String filter = "RestartFlowFactoryAction") { return buildActions.findAll { it.getClass().toString().contains(filter) } } String getRestartedBuildData(String jobName, int buildNumber) { def buildActions = getBuildActions(jobName, buildNumber) def filteredAction = filterBuildAction(buildActions, "RestartFlowFactoryAction") if (filteredAction.size() == 1) { return filteredAction[0].getOriginRunId() } else { return null } } void copyArtifactsFromRestartedBuild(String jobName, String buildNumber, String fileFilter) { String restartedBuildData = getRestartedBuildData(jobName, buildNumber as Integer) String restartedJobName = restartedBuildData.split("#")[0] String restartedBuildNumber = restartedBuildData.split("#")[-1] copyArtifacts(filter: fileFilter, projectName: restartedJobName, selector: context.specific("${restartedBuildNumber}")) }
Mateusz Filipek
added a comment - My version of workaround built on solution from gabih . Because stash can preserve to maximum of 50 builds I've prepared it on Archive artifacts and additionally it doesn't keep a stash in memory. It also requires Copy Artifact plugin - https://plugins.jenkins.io/copyartifact/
def withCheck( String blockName, Closure closure) {
script {
def buildStage = true
catchError(message: 'check previous build status' , stageResult: 'SUCCESS' , buildResult: 'SUCCESS' ) {
copyArtifactsFromRestartedBuild(context.env.JOB_NAME, context.env.BUILD_NUMBER, "stageStates/${blockName}" )
buildStage = false
}
if (buildStage) {
closure.call()
writeFile file: "${blockName}" , text: "1"
archiveArtifacts artifacts: "stageStates/${blockName}"
}
}
}
def getBuildActions( String jobName, int buildNumber) {
return Jenkins.instance.getItemByFullName(jobName).getBuildByNumber(buildNumber).properties.actions
}
def filterBuildAction(def buildActions, String filter = "RestartFlowFactoryAction" ) {
return buildActions.findAll { it.getClass().toString().contains(filter) }
}
String getRestartedBuildData( String jobName, int buildNumber) {
def buildActions = getBuildActions(jobName, buildNumber)
def filteredAction = filterBuildAction(buildActions, "RestartFlowFactoryAction" )
if (filteredAction.size() == 1) {
return filteredAction[0].getOriginRunId()
} else {
return null
}
}
void copyArtifactsFromRestartedBuild( String jobName, String buildNumber, String fileFilter) {
String restartedBuildData = getRestartedBuildData(jobName, buildNumber as Integer )
String restartedJobName = restartedBuildData.split( "#" )[0]
String restartedBuildNumber = restartedBuildData.split( "#" )[-1]
copyArtifacts(filter: fileFilter, projectName: restartedJobName, selector: context.specific( "${restartedBuildNumber}" ))
}
We'd love to see support for this. Are there any known workarounds that would allow us to make this work, other than creating a separate pipeline for each parallel branch?