Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64678

Support Shelved Jenkinsfile (mainly for Swarm reviews)

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: p4-plugin
    • Environment:
    • Similar Issues:

      Description

      We are introducing Jenkinsfile into an existing Perforce-hosted project.

      We have set it up as a Swarm project, which is correctly showing branches and Reviews.

      We have created a Multibranch Pipeline project, with the Helix Swarm Branch Source, as we want to have our Swarm Reviews run CI in a similar manner to Merge Requests in our GitLab installation.

      This project is configured for mode "by Jenkinsfile", with Script Path "Jenkinsfile".

      We have no Jenkinsfiles in our codebase yet, and one open review which is introducing the Jenkinsfile.

      However, the multibranch scan is providing the wrong reference to fetch the Jenkinsfile from the review.

      Our Scan Multibranch Pipeline Log looks like:

      Started by user Paul Hampson
      [Thu Jan 21 13:19:54 AEDT 2021] Starting branch indexing...
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/trunk/...@733645,734645 +
      Scanning for //ps/unreal/trunk/Jenkinsfile
      ... p4 files -e //ps/unreal/trunk/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/project-shooter-game/...@733645,734645 +
      Scanning for //ps/unreal/project-shooter-game/Jenkinsfile
      ... p4 files -e //ps/unreal/project-shooter-game/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      ... p4 counter change +
      ... p4 counter change +
      ... p4 changes -m1 -ssubmitted //jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123/...@7___ +
      ... p4 changes -m1 //ps/unreal/unreal-shooter-game/...@733645,734645 +
      Scanning for //ps/unreal/unreal-shooter-game/Jenkinsfile
      ... p4 files -e //ps/unreal/unreal-shooter-game/Jenkinsfile +
            ‘Jenkinsfile’ not found
      ... p4 login -s +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 info +
      ... p4 info +
      ... p4 client -o jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      ... p4 client -i +
      ...   View: +
      Scanning for //ps/unreal/trunk/Jenkinsfile@734627
      ... p4 files -e //ps/unreal/trunk/Jenkinsfile@734627 +
            ‘Jenkinsfile’ not found
      ... p4 client -d jenkinsTemp-194b52ab-489f-473b-9fa2-b48aedce7123 +
      [Thu Jan 21 13:19:54 AEDT 2021] Finished branch indexing. Indexing took 0.47 sec
      Finished: SUCCESS
      

      i.e. for Review 734267, it is attempting to fetch //ps/unreal/trunk/Jenkinsfile@734627 but being a Review (and hence a Shelf) it should be fetching //ps/unreal/trunk/Jenkinsfile@=734627

      I have confirmed through our Swarm instance that https://swarm.bigworldtech.com/files/ps/unreal/trunk/Jenkinsfile@=734627 exists, but https://swarm.bigworldtech.com/files/ps/unreal/trunk/Jenkinsfile@734627 does not.

      A log capture for org.jeninsci.plugins.p4 shows

      Jan 21, 2021 1:19:54 PM FINE org.jenkinsci.plugins.p4.scm.AbstractP4ScmSource
      SCM: retrieve Head: P4SCMHead: 734627 (//ps/unreal/trunk@734627)
      

      and I'm reasonably sure that this is the issue, and that it should simply be //ps/unreal/trunk@=734627.

      However, I acknowledge that perhaps this is more than just a missing =, if it turns out the workflow we need for Swarm Reviews is "checkout base and then unshelve"?

      Based on other reports in this JIRA, e.g., JENKINS-60220, I'm assuming that this problem is localised to the "by Jenkinsfile" part of the flow, and that once a Jenkinsfile is found, the actual workspace would correctly contain the Review's shelved state.

      It'd be good to know of any workarounds that might help here.

        Attachments

          Issue Links

            Activity

            Hide
            p_hampson Paul "TBBle" Hampson added a comment - - edited

            I am guessing that this issue means that Swarm Reviews with a modified Jenkinsfile would also be faulty, as they would build with the existing Jenkinsfile, not the modified one in the review.

            Thinking about it some more, I suspect the necessary behaviour is trying for the Jenkinsfile with the @=<reviewID>, and then falling back to the current @reviewID, as if the Jenkinsfile has not been modified, the former won't return it.

            Show
            p_hampson Paul "TBBle" Hampson added a comment - - edited I am guessing that this issue means that Swarm Reviews with a modified Jenkinsfile would also be faulty, as they would build with the existing Jenkinsfile, not the modified one in the review. Thinking about it some more, I suspect the necessary behaviour is trying for the Jenkinsfile with the @=<reviewID> , and then falling back to the current @reviewID , as if the Jenkinsfile has not been modified, the former won't return it.
            Hide
            p4karl Karl Wirth added a comment -

            We do not currently support Jenkinsfile in the review. It is assumed that the review is using an existing Jenkinsfile and building based on that logic.

            This is not documented anywhere so I have raised a doc enhancement request (JENKINS-64744) to clarify current behavior.

             

            Reproduction steps (proof that shelved Jenkinsfile is not used):

            (1) Keep it simple to start with just create a Pipeline from SCM job get Jenkinsfile from '//depot/ReviewBuild/main/...'

            (2) Submit some files and following Jenkinsfile to '//depot/ReviewBuild/main/...':

             

            pipeline {
            	agent { label 'master' }
            	stages {
            		stage("Repro") {
            			steps {
            				echo "P4_CHANGELIST is NOW:"
            				echo env.P4_CHANGELIST
            			}
            		}
            	}
            }
            
            

            (3) Create a new pending changelist and modify any of the files plus change the Jenkinsfile to:

             

             

            pipeline {
            	agent { label 'master' }
            	stages {
            		stage("Repro") {
            			steps {
            				echo "P4_CHANGELIST is NOW:"
            				echo env.P4_CHANGELIST
            				echo "This is the shelved code"
            			}
            		}
            	}
            }
            

            (4) Shelve the changelist:

             

             

            p4 shelve -c 18
            

            (5) Use the manual 'Build Review' option and specify the review (shelved) changelist and the 'status' of 'shelved':

             

            http://JENKINS_URL:8080/job/JOB_NAME/review/

             

            Result:

            This is the output seen in the console. Note that the text "This is the shelved code" does not appear:

             

            P4 Task: syncing files at change: 17
            ... p4 sync -q /var/lib/jenkins/workspace/ReviewBuildPipeline/...@17 +duration: (3ms)
            
            P4 Task: unshelve review: 18
            ... p4 unshelve -f -s18 -cdefault +
            //depot/ReviewBuild/main/Jenkinsfile
            //depot/ReviewBuild/main/test.txt
            ... duration: (21ms)
            P4: saving built changes.
            Found last change 17 on syncID jenkins-NODE_NAME-ReviewBuildPipeline-EXECUTOR_NUMBER
            ... p4 login -s +
            ... p4 client -o jenkins-master-ReviewBuildPipeline-0 +
            ... p4 info +
            ... p4 info +
            ... p4 client -o jenkins-master-ReviewBuildPipeline-0 +
            ... p4 change -o 18 +
            ... p4 user -o super +
            ... p4 describe -s -S 18 +
            ... p4 fixes -c18 +
            ... done
            
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] withEnv
            [Pipeline] {
            [Pipeline] stage
            [Pipeline] { (Repro)
            [Pipeline] echo
            P4_CHANGELIST is NOW:
            [Pipeline] echo
            17
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            Finished: SUCCESS
            

             

            As mentioned by Paul "TBBle" Hampson we would need to look for '@=' for the review changelist if the job was triggered by the review step and fall back to submitted Jenkinsfile if not found.

             

            For multibranch it becomes more complex because the job does not even exist yet so it would need to be created because we use 'Jenkinsfile' in the top level folder to define that the item is a branch. Would need to be similar to the 'Swarm Review' job creation we currently do.

            Show
            p4karl Karl Wirth added a comment - We do not currently support Jenkinsfile in the review. It is assumed that the review is using an existing Jenkinsfile and building based on that logic. This is not documented anywhere so I have raised a doc enhancement request ( JENKINS-64744 ) to clarify current behavior.   Reproduction steps (proof that shelved Jenkinsfile is not used): (1) Keep it simple to start with just create a Pipeline from SCM job get Jenkinsfile from '//depot/ReviewBuild/main/...' (2) Submit some files and following Jenkinsfile to '//depot/ReviewBuild/main/...':   pipeline { agent { label 'master' } stages { stage( "Repro" ) { steps { echo "P4_CHANGELIST is NOW:" echo env.P4_CHANGELIST } } } } (3) Create a new pending changelist and modify any of the files plus change the Jenkinsfile to:     pipeline { agent { label 'master' } stages { stage( "Repro" ) { steps { echo "P4_CHANGELIST is NOW:" echo env.P4_CHANGELIST echo "This is the shelved code" } } } } (4) Shelve the changelist:     p4 shelve -c 18 (5) Use the manual 'Build Review' option and specify the review (shelved) changelist and the 'status' of 'shelved':   http: //JENKINS_URL:8080/job/JOB_NAME/review/   Result: This is the output seen in the console. Note that the text "This is the shelved code" does not appear:   P4 Task: syncing files at change: 17 ... p4 sync -q / var /lib/jenkins/workspace/ReviewBuildPipeline/...@17 +duration: (3ms) P4 Task: unshelve review: 18 ... p4 unshelve -f -s18 -cdefault + //depot/ReviewBuild/main/Jenkinsfile //depot/ReviewBuild/main/test.txt ... duration: (21ms) P4: saving built changes. Found last change 17 on syncID jenkins-NODE_NAME-ReviewBuildPipeline-EXECUTOR_NUMBER ... p4 login -s + ... p4 client -o jenkins-master-ReviewBuildPipeline-0 + ... p4 info + ... p4 info + ... p4 client -o jenkins-master-ReviewBuildPipeline-0 + ... p4 change -o 18 + ... p4 user -o super + ... p4 describe -s -S 18 + ... p4 fixes -c18 + ... done [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Repro) [Pipeline] echo P4_CHANGELIST is NOW: [Pipeline] echo 17 [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS   As mentioned by Paul "TBBle" Hampson we would need to look for '@=' for the review changelist if the job was triggered by the review step and fall back to submitted Jenkinsfile if not found.   For multibranch it becomes more complex because the job does not even exist yet so it would need to be created because we use 'Jenkinsfile' in the top level folder to define that the item is a branch. Would need to be similar to the 'Swarm Review' job creation we currently do.
            Hide
            p_hampson Paul "TBBle" Hampson added a comment - - edited

            For my use-case, multibranch with the Helix Swarm source, it was correctly seeing that there was a review when it scanned, see "Scanning for //ps/unreal/trunk/Jenkinsfile@734627" in the pipeline log, so I expect that had it found the shelved Jenkinsfile, it would have created a job for it and proceeded to build correctly.

            I haven't looked at the code, but I suspect the Pipeline - From SCM support for reviews containing Jenkinsfile will be different from the Multibranch Pipeline - Helix Swarm Source support for reviews containing a JenkinsFile, as the latter has left it entirely up to the plugin's branch source to deliver a JenkinsFile (and a branch or review to go with it), but I suspect the former goes through a specific SCM API for fetching single files, and so may not have access to the "This is a review" details it needs to know.

             

            Since I raised this for Multibranch Pipeline - Helix Swarm, perhaps the Pipeline - From SCM should be a different issue? Unless I'm incorrect, and the same fix would resolve both systems together.

            Show
            p_hampson Paul "TBBle" Hampson added a comment - - edited For my use-case, multibranch with the Helix Swarm source, it was correctly seeing that there was a review when it scanned, see "Scanning for //ps/unreal/trunk/Jenkinsfile@734627" in the pipeline log, so I expect that had it found the shelved Jenkinsfile, it would have created a job for it and proceeded to build correctly. I haven't looked at the code, but I suspect the Pipeline - From SCM support for reviews containing Jenkinsfile will be different from the Multibranch Pipeline - Helix Swarm Source support for reviews containing a JenkinsFile, as the latter has left it entirely up to the plugin's branch source to deliver a JenkinsFile (and a branch or review to go with it), but I suspect the former goes through a specific SCM API for fetching single files, and so may not have access to the "This is a review" details it needs to know.   Since I raised this for Multibranch Pipeline - Helix Swarm, perhaps the Pipeline - From SCM should be a different issue? Unless I'm incorrect, and the same fix would resolve both systems together.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              p_hampson Paul "TBBle" Hampson
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: