Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72131

p4unshelve doesn't do appropriate variable expansion (like checkout scm: perforce() does)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • p4-plugin
    • Jenkins running on Ubuntu 22.04 executing on remote node via SSH. Jenkins Plugin version 1.14.2

      pipeline
      {
          agent {
              label "linux"
          }
          stages {
              stage("Checkout") {
                  steps {
                      checkout changelog: false, poll: false, scm: perforce(
                          credential: 'Perforce-Credentials',
                          populate: autoClean(delete: true, modtime: false, parallel: [enable: true, minbytes: '1024', minfiles: '1', threads: '4'], pin: null, quiet: true, replace: true, tidy: false),
                          workspace: manualSpec(
                              name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}",
                              spec: clientSpec(
                                  view: '''//path/abc/src/... //${P4_CLIENT}/...
                                           //path/def/src/... //${P4_CLIENT}/def/...'''
                              )
                          )
                      )
      
                      p4unshelve (resolve: 'ay', shelf: '316076', credential: 'Perforce-Credentials', tidy: true, ignoreEmpty: true,
                          workspace: manualSpec(
                              name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}",
                              spec: clientSpec(
                                  view: '''//path/abc/src/... //${P4_CLIENT}/...
                                           //path/def/src/... //${P4_CLIENT}/def/...'''
                              )
                          )
                      )
                  }
              }
          }
      }
      

       

      This will run until the p4unshelve step, then it will throw a stacktrace:

      P4: Unable to setup workspace: com.perforce.p4java.exception.RequestException: Error in client specification.
      Mapping '//P4_CLIENT/...' is not under '//jenkins-node-1-Test-0/...'.
      
      	at com.perforce.p4java.impl.mapbased.server.cmd.ResultMapParser.handleErrorStr(ResultMapParser.java:146)
      	at com.perforce.p4java.impl.mapbased.server.cmd.ResultMapParser.parseCommandResultMapIfIsInfoMessageAsString(ResultMapParser.java:98)
      	at com.perforce.p4java.impl.mapbased.server.cmd.ClientDelegator.updateClient(ClientDelegator.java:212)
      	at com.perforce.p4java.impl.mapbased.server.Server.updateClient(Server.java:1728)
      	at com.perforce.p4java.impl.mapbased.client.Client.update(Client.java:528)
      	at org.jenkinsci.plugins.p4.client.ClientHelper.updateClient(ClientHelper.java:191)
      	at org.jenkinsci.plugins.p4.client.ClientHelper.clientLogin(ClientHelper.java:170)
      	at org.jenkinsci.plugins.p4.client.ClientHelper.<init>(ClientHelper.java:104)
      	at org.jenkinsci.plugins.p4.unshelve.UnshelveBuilder.updateChangeLog(UnshelveBuilder.java:141)
      	at org.jenkinsci.plugins.p4.unshelve.UnshelveBuilder.unshelve(UnshelveBuilder.java:134)
      	at org.jenkinsci.plugins.p4.unshelve.UnshelveBuilderStep.perform(UnshelveBuilderStep.java:42)
      	at org.jenkinsci.plugins.p4.workflow.P4UnshelveStep$P4UnshelveStepExecution.run(P4UnshelveStep.java:133)
      	at org.jenkinsci.plugins.p4.workflow.P4UnshelveStep$P4UnshelveStepExecution.run(P4UnshelveStep.java:119)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      

       

      This bug maybe a duplicate of JENKINS-70584

          [JENKINS-72131] p4unshelve doesn't do appropriate variable expansion (like checkout scm: perforce() does)

          Eivind added a comment -

          I also note that the manual according to Helix says to use ''' instead of """ to prevent early variable expansion which may break the build.

          It is also unclear to me why I need to pass the workspace and credentials to the p4unshelve, the intention here is to really just sync the workspace and unshelve a change in it. Maybe this could have been better implemented using the checkout scm: perforce clause by adding another parameter "review" in addition to "change" (which can be used to pin the build to a specific change number or the latest by specifying "now"). Perhaps the documentation of the p4unshelve can be improved? 

          Perhaps the entire thing would be better implemented differently, e.g. p4.run() but that require one to permit dangerous functions signatures ...

          Eivind added a comment - I also note that the manual according to Helix says to use ''' instead of """ to prevent early variable expansion which may break the build. https://www.perforce.com/manuals/jenkins/Content/P4Jenkins/build-step-unshelve-snippet-generator.html?Highlight=p4%20unshelve%20jenkins It is also unclear to me why I need to pass the workspace and credentials to the p4unshelve, the intention here is to really just sync the workspace and unshelve a change in it. Maybe this could have been better implemented using the checkout scm: perforce clause by adding another parameter "review" in addition to "change" (which can be used to pin the build to a specific change number or the latest by specifying "now"). Perhaps the documentation of the p4unshelve can be improved?  Perhaps the entire thing would be better implemented differently, e.g. p4.run() but that require one to permit dangerous functions signatures ...

          Eivind added a comment -
          pipeline 
          {
              agent {
                  label "linux"
              }
              environment {
                  P4_REVIEW = "316076"
              }
              stages {
                  stage("Setup") {
                      steps {
                          script {
                              checkout changelog: false, poll: false, scm: perforce(
                                  credential: 'Perforce-Credentials',
                                  populate: autoClean(delete: true, modtime: false, parallel: [enable: true, minbytes: '1024', minfiles: '1', threads: '4'], pin: null, quiet: true, replace: true, tidy: true),
                                  workspace: manualSpec(
                                      name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}",
                                      spec: clientSpec(
                                          view: '''//path/abc/src/... //${P4_CLIENT}/...
                                                   //path/def/... //${P4_CLIENT}/def/...'''
                                      )
                                  )
                              )
                              if (env.P4_REVIEW != null) {
                                  p4unshelve shelf: env.P4_REVIEW
                              }
                          }
                       }
                    }
                }
          }
          

          Somehow sticking the p4unshelve into a script block and calling it with just the shelf: "xxx" parameter set seems to work for me, and it doesn't give me any fuzz.

          Is this even intentional or a supported way of doing the p4unshelve? If so, it probably should be documented.

          Eivind added a comment - pipeline  {     agent {         label "linux"     }     environment {         P4_REVIEW = "316076"     }     stages {         stage( "Setup" ) {             steps {                 script {                     checkout changelog: false , poll: false , scm: perforce(                         credential: 'Perforce-Credentials' ,                         populate: autoClean(delete: true , modtime: false , parallel: [enable: true , minbytes: '1024' , minfiles: '1' , threads: '4' ], pin: null , quiet: true , replace: true , tidy: true ),                         workspace: manualSpec(                             name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" ,                             spec: clientSpec(                                 view: ''' //path/abc/src/... //${P4_CLIENT}/...                                           //path/ def /... //${P4_CLIENT}/ def /...'''                             )                         )                     )                     if (env.P4_REVIEW != null ) {                         p4unshelve shelf: env.P4_REVIEW                     }                 } } } } } Somehow sticking the p4unshelve into a script block and calling it with just the shelf: "xxx" parameter set seems to work for me, and it doesn't give me any fuzz. Is this even intentional or a supported way of doing the p4unshelve? If so, it probably should be documented.

          Eivind added a comment -

          While my last comment seem to have done the trick, it seems unreliable when used in a parallel stage on multiple different nodes. I somehow ended up with an p4 unshelve on a different path (the path of the client spec on my Mac appeared on one of my Linux nodes). Ultimately this seems racy ...

          Eivind added a comment - While my last comment seem to have done the trick, it seems unreliable when used in a parallel stage on multiple different nodes. I somehow ended up with an p4 unshelve on a different path (the path of the client spec on my Mac appeared on one of my Linux nodes). Ultimately this seems racy ...

          Karl Wirth added a comment -

          HI enaess (and P4Jenkins developers) - Not being able to trust P4_CLIENT is this known bug:

             https://issues.jenkins.io/browse/JENKINS-60213

          The problem is Jenkins gives us the wrong information for the current context:

             https://issues.jenkins.io/browse/JENKINS-48882

          Karl Wirth added a comment - HI enaess (and P4Jenkins developers) - Not being able to trust P4_CLIENT is this known bug:     https://issues.jenkins.io/browse/JENKINS-60213 The problem is Jenkins gives us the wrong information for the current context:     https://issues.jenkins.io/browse/JENKINS-48882

          Eivind added a comment -

          p4karl - Regarding the executor number, you maybe right; but I've configured my nodes to only execute a job at time, i.e. there is only one executor and that's the current one. I could simply remove that from the P4 client name, but it shouldn't matter for this problem. 

          The bug here being:

          checkout with scm: perforce, the clientSpec's view is properly substituted with ${P4_CLIENT}

          p4unshelve does specify the view the same way, but variable substitution does not work. I'd have to use " quotes to have groovy do it instead of the recommended late stage variable interpolation according to your docs, or do the string replace myself.

           


           

          The fact I have to supply both these checkout and p4unshelve with a separate client spec is annoying, in part because I have to do it in 4-5 other places in the same pipeline.

          Eivind added a comment - p4karl - Regarding the executor number, you maybe right; but I've configured my nodes to only execute a job at time, i.e. there is only one executor and that's the current one. I could simply remove that from the P4 client name, but it shouldn't matter for this problem.  The bug here being: checkout with scm: perforce, the clientSpec's view is properly substituted with ${P4_CLIENT} p4unshelve does specify the view the same way, but variable substitution does not work. I'd have to use " quotes to have groovy do it instead of the recommended late stage variable interpolation according to your docs, or do the string replace myself.     The fact I have to supply both these checkout and p4unshelve with a separate client spec is annoying, in part because I have to do it in 4-5 other places in the same pipeline.

          Karl Wirth added a comment -

          Hi enaess,

          Thanks. Confirmed the variable expansion is definitely strange here. If I use double quotes in the unshelve it works but double quotes do not work for multiline so it's not a valid workaround.

                          p4unshelve (resolve: 'ay', shelf: '316076', credential: 'Perforce-Credentials', tidy: true, ignoreEmpty: true,
                              workspace: manualSpec(
                                  name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}",
                                  spec: clientSpec(
                                      view: "//path/abc/src/... //${P4_CLIENT}/..."
                                  )
                              )
                          ) 

          Bug confirmed and notifying the devs.

           

          Karl Wirth added a comment - Hi enaess , Thanks. Confirmed the variable expansion is definitely strange here. If I use double quotes in the unshelve it works but double quotes do not work for multiline so it's not a valid workaround. p4unshelve (resolve: 'ay' , shelf: '316076' , credential: 'Perforce-Credentials' , tidy: true , ignoreEmpty: true , workspace: manualSpec( name: "${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" , spec: clientSpec( view: " //path/abc/src/... //${P4_CLIENT}/..." ) ) ) Bug confirmed and notifying the devs.  

            Unassigned Unassigned
            enaess Eivind
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: