• Declarative backlog

      This issue is based on JENKINS-45455 which solves the issue of restarting a Jenkins Pipeline from Top-Level stages.

      Description
      When building on multiple platforms, the builds are usually ran in parallel. We should be able to replay the build of a single platform. (e.g. if the Win64 build fails, we do not want to retrigger the already successful Macos and Linux builds)
      In a multiplatform build pipeline, the steps taking the most time are the "build" steps themselves and they are running in parallel to get fast build times. Restarting from a top level in that context boils down to re-running the whole job.

      Use cases

      • Restarting a Jenkins Pipeline from a sub-stage or parallel stage
      • Given the following graph, we should be able to replay from any stage (e.g. Build-Win64, Test-MacOs, Deploy) :
        Build-Win64 -> Test-Win64 -| 
        Build-Win32 -> Test-Win32 -|-> Deploy
        Build-MacOS -> Test-MacOS -|
        Build-Linux -> Test-Linux -|

         

          [JENKINS-52391] Restarting Parallel Stages

          Adam Kapos added a comment -

          We'd love to see support for this. Are there any known workarounds that would allow us to make this work, other than creating a separate pipeline for each parallel branch?

          Adam Kapos added a comment - We'd love to see support for this. Are there any known workarounds that would allow us to make this work, other than creating a separate pipeline for each parallel branch?

          Gabriel Herisanu added a comment - - edited

          You can use something like this:

          #!groovy
          def withCheck(String blockName, Closure closure) {
              script {
                  def buildStage = true
                  catchError(message: 'check previous build status', stageResult:'SUCCESS', buildResult: 'SUCCESS') {
                      unstash name:"${blockName}"
                      buildStage = false
                  }
          
                  if (buildStage) {
                      closure.call()
                      writeFile file: "${blockName}", text: "1"
                      stash name: "${blockName}", includes: "${blockName}"
                  }
              }
          }
          
          pipeline {
              agent none
          
              options {
                  preserveStashes()
              }
          
              stages {
                  stage("Build and test") {
                      parallel() {
                          stage("Build/Test Win64") {
                              agent {
                                  label 'master'
                              }
                              steps {
                                  withCheck("build-deploy-Win64") {
                                      echo "test"
                                  }
          
                                  withCheck("test-Win64") {
                                      echo "test"
                                  }
                              }
                          }
          
                          stage("Build/Test Win32") {
                              agent {
                                  label 'master'
                              }
                              steps {
                                  withCheck("build-deploy-Win32") {
                                      echo "test"
                                  }
          
                                  withCheck("test-Win32") {
                                      echo "test"
                                  }
                              }
                          }
                      }
                  }
          
                  stage("Deploy") {
                      agent {
                          label 'master'
                      }
                      steps {
                          withCheck("build-deploy-win64") {
                              echo "test"
                          }
          
                          withCheck("test-win64") {
                              echo "test"
                          }
                      }
                  }
              }
          }
          

           

          Gabriel Herisanu added a comment - - edited You can use something like this: #!groovy def withCheck( String blockName, Closure closure) { script { def buildStage = true catchError(message: 'check previous build status' , stageResult: 'SUCCESS' , buildResult: 'SUCCESS' ) { unstash name: "${blockName}" buildStage = false } if (buildStage) { closure.call() writeFile file: "${blockName}" , text: "1" stash name: "${blockName}" , includes: "${blockName}" } } } pipeline { agent none options { preserveStashes() } stages { stage( "Build and test" ) { parallel() { stage( "Build/Test Win64" ) { agent { label 'master' } steps { withCheck( "build-deploy-Win64" ) { echo "test" } withCheck( "test-Win64" ) { echo "test" } } } stage( "Build/Test Win32" ) { agent { label 'master' } steps { withCheck( "build-deploy-Win32" ) { echo "test" } withCheck( "test-Win32" ) { echo "test" } } } } } stage( "Deploy" ) { agent { label 'master' } steps { withCheck( "build-deploy-win64" ) { echo "test" } withCheck( "test-win64" ) { echo "test" } } } } }  

          Joan Touzet added a comment -

          gabih This bug is about declarative language features - see the Epic Link and Sprint tagging above.

          Your suggestion isn't declarative.

          Joan Touzet added a comment - gabih This bug is about declarative language features - see the Epic Link and Sprint tagging above. Your suggestion isn't declarative.

          gabih That actually looks like a neat trick. Will check whether we will be able to use it in pure scripted pipeline

          Joerg Schwaerzler added a comment - gabih That actually looks like a neat trick. Will check whether we will be able to use it in pure scripted pipeline

          gabih Can you confirm that the preserveStashes option will be honored by any stash/unstash called from within the scripted block? Currently it seems like those stashes are not preserved on our Jenkins instance when calling stash from within the script block. But maybe we just need an update of Jenkins or some plugin...

          Joerg Schwaerzler added a comment - gabih Can you confirm that the preserveStashes option will be honored by any stash/unstash called from within the scripted block? Currently it seems like those stashes are not preserved on our Jenkins instance when calling stash from within the script block. But maybe we just need an update of Jenkins or some plugin...

          macdrega preserveStashes(/buildCount: 5/) works if the pipeline is started form the same Jenkins instance

          Gabriel Herisanu added a comment - macdrega  preserveStashes(/ buildCount: 5 /) works if the pipeline is started form the same Jenkins instance

          What I did not realize is that preserverStashes will only work when restarting from some stage... Looks better now - thanks.

          Joerg Schwaerzler added a comment - What I did not realize is that preserverStashes will only work when restarting from some stage... Looks better now - thanks.

          Zak Taccardi added a comment -

          Is the matrix feature blocked by a lack of support for restarting here?

          Zak Taccardi added a comment - Is the matrix feature blocked by a lack of support for restarting here?

          My version of workaround built on solution from gabih. Because stash can preserve to maximum of 50 builds I've prepared it on Archive artifacts and additionally it doesn't keep a stash in memory. It also requires Copy Artifact plugin - https://plugins.jenkins.io/copyartifact/ 

           def withCheck(String blockName, Closure closure) {
              script {
                  def buildStage = true
                  catchError(message: 'check previous build status', stageResult:'SUCCESS', buildResult: 'SUCCESS') {
                      copyArtifactsFromRestartedBuild(context.env.JOB_NAME, context.env.BUILD_NUMBER, "stageStates/${blockName}")
                      buildStage = false
                  }
          
                  if (buildStage) {
                      closure.call()
                      writeFile file: "${blockName}", text: "1"
                      archiveArtifacts artifacts: "stageStates/${blockName}"
                  }
              }
          }
          
          def getBuildActions(String jobName, int buildNumber) {
             return Jenkins.instance.getItemByFullName(jobName).getBuildByNumber(buildNumber).properties.actions
          }
          
          def filterBuildAction(def buildActions, String filter = "RestartFlowFactoryAction") {
             return buildActions.findAll { it.getClass().toString().contains(filter) }
          }
          
          String getRestartedBuildData(String jobName, int buildNumber) {
             def buildActions = getBuildActions(jobName, buildNumber)
             def filteredAction = filterBuildAction(buildActions, "RestartFlowFactoryAction")
             if (filteredAction.size() == 1) {
                return filteredAction[0].getOriginRunId()
             } else {
                return null
             }
          }
          
          void copyArtifactsFromRestartedBuild(String jobName, String buildNumber, String fileFilter) {
             String restartedBuildData = getRestartedBuildData(jobName, buildNumber as Integer)
             String restartedJobName = restartedBuildData.split("#")[0]
             String restartedBuildNumber = restartedBuildData.split("#")[-1]
             copyArtifacts(filter: fileFilter, projectName: restartedJobName, selector: context.specific("${restartedBuildNumber}"))
          }

           

          Mateusz Filipek added a comment - My version of workaround built on solution from gabih . Because stash can preserve to maximum of 50 builds I've prepared it on Archive artifacts and additionally it doesn't keep a stash in memory. It also requires Copy Artifact plugin - https://plugins.jenkins.io/copyartifact/   def withCheck( String blockName, Closure closure) { script { def buildStage = true catchError(message: 'check previous build status' , stageResult: 'SUCCESS' , buildResult: 'SUCCESS' ) { copyArtifactsFromRestartedBuild(context.env.JOB_NAME, context.env.BUILD_NUMBER, "stageStates/${blockName}" ) buildStage = false } if (buildStage) { closure.call() writeFile file: "${blockName}" , text: "1" archiveArtifacts artifacts: "stageStates/${blockName}" } } } def getBuildActions( String jobName, int buildNumber) { return Jenkins.instance.getItemByFullName(jobName).getBuildByNumber(buildNumber).properties.actions } def filterBuildAction(def buildActions, String filter = "RestartFlowFactoryAction" ) { return buildActions.findAll { it.getClass().toString().contains(filter) } } String getRestartedBuildData( String jobName, int buildNumber) { def buildActions = getBuildActions(jobName, buildNumber) def filteredAction = filterBuildAction(buildActions, "RestartFlowFactoryAction" ) if (filteredAction.size() == 1) { return filteredAction[0].getOriginRunId() } else { return null } } void copyArtifactsFromRestartedBuild( String jobName, String buildNumber, String fileFilter) { String restartedBuildData = getRestartedBuildData(jobName, buildNumber as Integer ) String restartedJobName = restartedBuildData.split( "#" )[0] String restartedBuildNumber = restartedBuildData.split( "#" )[-1] copyArtifacts(filter: fileFilter, projectName: restartedJobName, selector: context.specific( "${restartedBuildNumber}" )) }  

            Unassigned Unassigned
            franknarf8_ni Frank Genois
            Votes:
            52 Vote for this issue
            Watchers:
            38 Start watching this issue

              Created:
              Updated: