Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64678

Support Shelved Jenkinsfile (mainly for Swarm reviews)

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin

      We are introducing Jenkinsfile into an existing Perforce-hosted project.

      We have set it up as a Swarm project, which is correctly showing branches and Reviews.

      We have created a Multibranch Pipeline project, with the Helix Swarm Branch Source, as we want to have our Swarm Reviews run CI in a similar manner to Merge Requests in our GitLab installation.

      This project is configured for mode "by Jenkinsfile", with Script Path "Jenkinsfile".

      We have no Jenkinsfiles in our codebase yet, and one open review which is introducing the Jenkinsfile.

      However, the multibranch scan is providing the wrong reference to fetch the Jenkinsfile from the review.

      Our Scan Multibranch Pipeline Log looks like:

      Started by user Paul Hampson
      [Thu Jan 21 13:19:54 AEDT 2021] Starting branch indexing...
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/trunk/...@733645,734645 +
      Scanning for //ps/unreal/trunk/Jenkinsfile
      ... p4 files -e //ps/unreal/trunk/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/project-shooter-game/...@733645,734645 +
      Scanning for //ps/unreal/project-shooter-game/Jenkinsfile
      ... p4 files -e //ps/unreal/project-shooter-game/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/unreal-shooter-game/...@733645,734645 +
      Scanning for //ps/unreal/unreal-shooter-game/Jenkinsfile
      ... p4 files -e //ps/unreal/unreal-shooter-game/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      Scanning for //ps/unreal/trunk/Jenkinsfile@734627
      ... p4 files -e //ps/unreal/trunk/Jenkinsfile@734627 +
            ‘Jenkinsfile’ not found
      ... p4 client -d jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      [Thu Jan 21 13:19:54 AEDT 2021] Finished branch indexing. Indexing took 0.47 sec
      Finished: SUCCESS
      

      i.e. for Review 734267, it is attempting to fetch //ps/unreal/trunk/Jenkinsfile@734627 but being a Review (and hence a Shelf) it should be fetching //ps/unreal/trunk/Jenkinsfile@=734627

      I have confirmed through our Swarm instance that https://swarm.bigworldtech.com/files/ps/unreal/trunk/Jenkinsfile@=734627 exists, but https://swarm.bigworldtech.com/files/ps/unreal/trunk/Jenkinsfile@734627 does not.

      A log capture for org.jeninsci.plugins.p4 shows

      Jan 21, 2021 1:19:54 PM FINE org.jenkinsci.plugins.p4.scm.AbstractP4ScmSource
      SCM: retrieve Head: P4SCMHead: 734627 (//ps/unreal/trunk@734627)
      

      and I'm reasonably sure that this is the issue, and that it should simply be //ps/unreal/trunk@=734627.

      However, I acknowledge that perhaps this is more than just a missing =, if it turns out the workflow we need for Swarm Reviews is "checkout base and then unshelve"?

      Based on other reports in this JIRA, e.g., JENKINS-60220, I'm assuming that this problem is localised to the "by Jenkinsfile" part of the flow, and that once a Jenkinsfile is found, the actual workspace would correctly contain the Review's shelved state.

      It'd be good to know of any workarounds that might help here.

          [JENKINS-64678] Support Shelved Jenkinsfile (mainly for Swarm reviews)

          Paul "TBBle" Hampson added a comment - - edited

          I am guessing that this issue means that Swarm Reviews with a modified Jenkinsfile would also be faulty, as they would build with the existing Jenkinsfile, not the modified one in the review.

          Thinking about it some more, I suspect the necessary behaviour is trying for the Jenkinsfile with the @=<reviewID>, and then falling back to the current @reviewID, as if the Jenkinsfile has not been modified, the former won't return it.

          Paul "TBBle" Hampson added a comment - - edited I am guessing that this issue means that Swarm Reviews with a modified Jenkinsfile would also be faulty, as they would build with the existing Jenkinsfile, not the modified one in the review. Thinking about it some more, I suspect the necessary behaviour is trying for the Jenkinsfile with the @=<reviewID> , and then falling back to the current @reviewID , as if the Jenkinsfile has not been modified, the former won't return it.

          Karl Wirth added a comment -

          We do not currently support Jenkinsfile in the review. It is assumed that the review is using an existing Jenkinsfile and building based on that logic.

          This is not documented anywhere so I have raised a doc enhancement request (JENKINS-64744) to clarify current behavior.

           

          Reproduction steps (proof that shelved Jenkinsfile is not used):

          (1) Keep it simple to start with just create a Pipeline from SCM job get Jenkinsfile from '//depot/ReviewBuild/main/...'

          (2) Submit some files and following Jenkinsfile to '//depot/ReviewBuild/main/...':

           

          pipeline {
          	agent { label 'master' }
          	stages {
          		stage("Repro") {
          			steps {
          				echo "P4_CHANGELIST is NOW:"
          				echo env.P4_CHANGELIST
          			}
          		}
          	}
          }
          
          

          (3) Create a new pending changelist and modify any of the files plus change the Jenkinsfile to:

           

           

          pipeline {
          	agent { label 'master' }
          	stages {
          		stage("Repro") {
          			steps {
          				echo "P4_CHANGELIST is NOW:"
          				echo env.P4_CHANGELIST
          				echo "This is the shelved code"
          			}
          		}
          	}
          }
          

          (4) Shelve the changelist:

           

           

          p4 shelve -c 18
          

          (5) Use the manual 'Build Review' option and specify the review (shelved) changelist and the 'status' of 'shelved':

           

          http://JENKINS_URL:8080/job/JOB_NAME/review/

           

          Result:

          This is the output seen in the console. Note that the text "This is the shelved code" does not appear:

           

          P4 Task: syncing files at change: 17
          ... p4 sync -q /var/lib/jenkins/workspace/ReviewBuildPipeline/...@17 +duration: (3ms)
          
          P4 Task: unshelve review: 18
          ... p4 unshelve -f -s18 -cdefault +
          //depot/ReviewBuild/main/Jenkinsfile
          //depot/ReviewBuild/main/test.txt
          ... duration: (21ms)
          P4: saving built changes.
          Found last change 17 on syncID jenkins-NODE_NAME-ReviewBuildPipeline-EXECUTOR_NUMBER
          ... p4 login -s +
          ... p4 client -o jenkins-master-ReviewBuildPipeline-0 +
          ... p4 info +
          ... p4 info +
          ... p4 client -o jenkins-master-ReviewBuildPipeline-0 +
          ... p4 change -o 18 +
          ... p4 user -o super +
          ... p4 describe -s -S 18 +
          ... p4 fixes -c18 +
          ... done
          
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] withEnv
          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Repro)
          [Pipeline] echo
          P4_CHANGELIST is NOW:
          [Pipeline] echo
          17
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          

           

          As mentioned by p_hampson we would need to look for '@=' for the review changelist if the job was triggered by the review step and fall back to submitted Jenkinsfile if not found.

           

          For multibranch it becomes more complex because the job does not even exist yet so it would need to be created because we use 'Jenkinsfile' in the top level folder to define that the item is a branch. Would need to be similar to the 'Swarm Review' job creation we currently do.

          Karl Wirth added a comment - We do not currently support Jenkinsfile in the review. It is assumed that the review is using an existing Jenkinsfile and building based on that logic. This is not documented anywhere so I have raised a doc enhancement request ( JENKINS-64744 ) to clarify current behavior.   Reproduction steps (proof that shelved Jenkinsfile is not used): (1) Keep it simple to start with just create a Pipeline from SCM job get Jenkinsfile from '//depot/ReviewBuild/main/...' (2) Submit some files and following Jenkinsfile to '//depot/ReviewBuild/main/...':   pipeline { agent { label 'master' } stages { stage( "Repro" ) { steps { echo "P4_CHANGELIST is NOW:" echo env.P4_CHANGELIST } } } } (3) Create a new pending changelist and modify any of the files plus change the Jenkinsfile to:     pipeline { agent { label 'master' } stages { stage( "Repro" ) { steps { echo "P4_CHANGELIST is NOW:" echo env.P4_CHANGELIST echo "This is the shelved code" } } } } (4) Shelve the changelist:     p4 shelve -c 18 (5) Use the manual 'Build Review' option and specify the review (shelved) changelist and the 'status' of 'shelved':   http: //JENKINS_URL:8080/job/JOB_NAME/review/   Result: This is the output seen in the console. Note that the text "This is the shelved code" does not appear:   P4 Task: syncing files at change: 17 ... p4 sync -q / var /lib/jenkins/workspace/ReviewBuildPipeline/...@17 +duration: (3ms) P4 Task: unshelve review: 18 ... p4 unshelve -f -s18 -cdefault + //depot/ReviewBuild/main/Jenkinsfile //depot/ReviewBuild/main/test.txt ... duration: (21ms) P4: saving built changes. Found last change 17 on syncID jenkins-NODE_NAME-ReviewBuildPipeline-EXECUTOR_NUMBER ... p4 login -s + ... p4 client -o jenkins-master-ReviewBuildPipeline-0 + ... p4 info + ... p4 info + ... p4 client -o jenkins-master-ReviewBuildPipeline-0 + ... p4 change -o 18 + ... p4 user -o super + ... p4 describe -s -S 18 + ... p4 fixes -c18 + ... done [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Repro) [Pipeline] echo P4_CHANGELIST is NOW: [Pipeline] echo 17 [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS   As mentioned by p_hampson we would need to look for '@=' for the review changelist if the job was triggered by the review step and fall back to submitted Jenkinsfile if not found.   For multibranch it becomes more complex because the job does not even exist yet so it would need to be created because we use 'Jenkinsfile' in the top level folder to define that the item is a branch. Would need to be similar to the 'Swarm Review' job creation we currently do.

          Paul "TBBle" Hampson added a comment - - edited

          For my use-case, multibranch with the Helix Swarm source, it was correctly seeing that there was a review when it scanned, see "Scanning for //ps/unreal/trunk/Jenkinsfile@734627" in the pipeline log, so I expect that had it found the shelved Jenkinsfile, it would have created a job for it and proceeded to build correctly.

          I haven't looked at the code, but I suspect the Pipeline - From SCM support for reviews containing Jenkinsfile will be different from the Multibranch Pipeline - Helix Swarm Source support for reviews containing a JenkinsFile, as the latter has left it entirely up to the plugin's branch source to deliver a JenkinsFile (and a branch or review to go with it), but I suspect the former goes through a specific SCM API for fetching single files, and so may not have access to the "This is a review" details it needs to know.

           

          Since I raised this for Multibranch Pipeline - Helix Swarm, perhaps the Pipeline - From SCM should be a different issue? Unless I'm incorrect, and the same fix would resolve both systems together.

          Paul "TBBle" Hampson added a comment - - edited For my use-case, multibranch with the Helix Swarm source, it was correctly seeing that there was a review when it scanned, see "Scanning for //ps/unreal/trunk/Jenkinsfile@734627" in the pipeline log, so I expect that had it found the shelved Jenkinsfile, it would have created a job for it and proceeded to build correctly. I haven't looked at the code, but I suspect the Pipeline - From SCM support for reviews containing Jenkinsfile will be different from the Multibranch Pipeline - Helix Swarm Source support for reviews containing a JenkinsFile, as the latter has left it entirely up to the plugin's branch source to deliver a JenkinsFile (and a branch or review to go with it), but I suspect the former goes through a specific SCM API for fetching single files, and so may not have access to the "This is a review" details it needs to know.   Since I raised this for Multibranch Pipeline - Helix Swarm, perhaps the Pipeline - From SCM should be a different issue? Unless I'm incorrect, and the same fix would resolve both systems together.

            Unassigned Unassigned
            p_hampson Paul "TBBle" Hampson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: