Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45999

Provide light weight checkout functionality for perforce in pipeline jobs

    XMLWordPrintable

Details

    Description

      Provide a light weight check out functionality for perforce so that when initializing the job or running it will not check out whole code repo to master. When we hold a huge code base it take too much time to trigger the build jobs as it has to chekcout the whole repo on to master and configure from jenkinsfile and then it starts it's build process.

       

      Must work for streams.

      Attachments

        Issue Links

          Activity

            rpetti Rob Petti added a comment -

            Karthik, perforce-plugin does not support pipeline at all. Are you sure you filed this against the correct plugin?

            rpetti Rob Petti added a comment - Karthik, perforce-plugin does not support pipeline at all. Are you sure you filed this against the correct plugin?
            karthik546 karthik paidi added a comment -

            rpetti yes rob we are using perforce and all of our pipeline jobs are configure with that 

            karthik546 karthik paidi added a comment - rpetti yes rob we are using perforce and all of our pipeline jobs are configure with that 
            rpetti Rob Petti added a comment -

            Karthik, the perforce-plugin is deprecated and does not support pipeline, so I cannot imagine how you are even using it. Are you absolutely sure you are using perforce-plugin and not p4-plugin? If you are, then I'll just close this ticket, because there's no way this will be implemented in perforce-plugin.

            rpetti Rob Petti added a comment - Karthik, the perforce-plugin is deprecated and does not support pipeline, so I cannot imagine how you are even using it. Are you absolutely sure you are using  perforce-plugin and not  p4-plugin ? If you are, then I'll just close this ticket, because there's no way this will be implemented in perforce-plugin.
            karthik546 karthik paidi added a comment - - edited

            rpetti sorry it's P4-plugin 

            karthik546 karthik paidi added a comment - - edited rpetti sorry it's P4-plugin 
            p4paul Paul Allen added a comment -

            Thank you Rob for forwarding this on.

            In the Job definition create a workspace view (or virtual stream) to map just the Jenkinsfile (plus any scripts/libs). Use a different name, I append '-script', then in the Jenkinsfile map your whole codebase in a different workspace (i.e the default).  

            https://github.com/jenkinsci/p4-plugin/blob/master/WORKFLOW.md#using-a-versioned-jenkinsfile

            p4paul Paul Allen added a comment - Thank you Rob for forwarding this on. In the Job definition create a workspace view (or virtual stream) to map just the Jenkinsfile (plus any scripts/libs). Use a different name, I append '-script', then in the Jenkinsfile map your whole codebase in a different workspace (i.e the default).   https://github.com/jenkinsci/p4-plugin/blob/master/WORKFLOW.md#using-a-versioned-jenkinsfile
            ynedelin yury nedelin added a comment - - edited

            Hi p4paul,

            I am having similar strange behavior with this plug-in.

            This is my Jenkinsfile

            pipeline {
                agent {
                    label 'master'
                }
                stages {
                    stage ('test') {
                        steps{
                            sh 'mkdir test'
                        }
                    }
                }
            }

            for p4 set up I copied Workflow Setup Guide exactly as you have it for Versioned Jenkinsfile, with jenkins-${JOB_NAME}-script for workspace name.

            Jenkins creates three directories in the workspace:  workspace/jobname, workspace/jobname@script and workspace/jobname@tmp.

            Looks like workspace/jobname is the one that 'mkdir test' runs in and creates a 'test' directory, it also gets a copy of Jenkinsfile.

            workspace/jobname@script only gets Jenkinsfile.

            workspace/jobname@tmp is empty.

            In p4 I now see a workspace named jenkins-jobname-script that's pointing to Users/jenkins/.jenkins/workspace/jobname

             

            Is this how it should work ?

             

            For another question, how can I change workspace root using p4 pipeline command for a particular p4 workspace ?

             

            Thanks,

            Yury

             

             

             

             

            ynedelin yury nedelin added a comment - - edited Hi p4paul , I am having similar strange behavior with this plug-in. This is my Jenkinsfile pipeline {     agent {         label 'master'     }     stages {         stage ('test') {             steps{                 sh 'mkdir test'             }         }     } } for p4 set up I copied Workflow Setup Guide exactly as you have it for Versioned Jenkinsfile, with jenkins-${JOB_NAME}-script for workspace name. Jenkins creates three directories in the workspace:  workspace/jobname, workspace/jobname@script and workspace/jobname@tmp. Looks like workspace/jobname is the one that 'mkdir test' runs in and creates a 'test' directory, it also gets a copy of Jenkinsfile. workspace/jobname@script only gets Jenkinsfile. workspace/jobname@tmp is empty. In p4 I now see a workspace named jenkins-jobname-script that's pointing to Users/jenkins/.jenkins/workspace/jobname   Is this how it should work ?   For another question, how can I change workspace root using p4 pipeline command for a particular p4 workspace ?   Thanks, Yury        
            p4paul Paul Allen added a comment -

            Hi Yury,

            Yes use a name like 'jenkins-${JOB_NAME}-script' for the Workspace that checks out the Jenkinsfile (although ${NODE_NAME} is important to include if you are using slaves).  I would recommend limiting the View so only the script (or scripts if required)  are checked out and not all your source and assets.  If you are running on a master/slave setup the Jenkinsfile will appear on the master in the @script dir (as you have already seen).

            Jenkins will then execute the groovy Jenkinsfile script, on the master or slave (as determined by the agent field), this in in the jobname dir (no @'s here).  Presumably you would have a checkout step in your Jenkinsfile to get all your source code/assets?  

            Use the `checkout` DSL (the Pipeline Snippet tool helps to generate the groovy code) or if your code paths are relatively simple (just a stream or single depot path) use the `p4sync` DSL.

            stage('Checkout...') {
              steps {
                p4sync charset: 'none', 
                  credential: 'someID', 
                  populate: [$class: 'ForceCleanImpl', have: false, pin: '', quiet: true], 
                  source: [$class: 'StreamSource', stream: '//streams/main']
              }
            }

            As for workspace roots; Jenkins will determine the workspace root for the job and the Perforce workspace root should be the same location.  The p4-plugin will overwrite the Perforce workspace for every build. I would recommend not changing the workspace root, but if you must use the Pipeline `ws`; look here for the details.

            p4paul Paul Allen added a comment - Hi Yury, Yes use a name like 'jenkins-${JOB_NAME}-script' for the Workspace that checks out the Jenkinsfile (although ${NODE_NAME} is important to include if you are using slaves).  I would recommend limiting the View so only the script (or scripts if required)  are checked out and not all your source and assets.  If you are running on a master/slave setup the Jenkinsfile will appear on the master in the @script dir (as you have already seen). Jenkins will then execute the groovy Jenkinsfile script, on the master or slave (as determined by the agent field), this in in the jobname dir (no @'s here).  Presumably you would have a checkout step in your Jenkinsfile to get all your source code/assets?   Use the `checkout` DSL (the Pipeline Snippet tool helps to generate the groovy code) or if your code paths are relatively simple (just a stream or single depot path) use the `p4sync` DSL. stage( 'Checkout...' ) { steps { p4sync charset: 'none' ,    credential: 'someID' ,    populate: [$class: 'ForceCleanImpl' , have: false , pin: '', quiet: true ],    source: [$class: 'StreamSource' , stream: ' //streams/main' ] } } As for workspace roots; Jenkins will determine the workspace root for the job and the Perforce workspace root should be the same location.  The p4-plugin will overwrite the Perforce workspace for every build. I would recommend not changing the workspace root, but if you must use the Pipeline `ws`; look here for the details.
            ynedelin yury nedelin added a comment -

            Hi Paul, thanks for your help, everything has been working fairly well for us in terms of getting the job done. I wanted to double check one thing, with pipeline method will p4 create two locations with full sync indicated by view mapping ? I see full set of files in MyRoot@script and also in MyRoot, is that how it should work? The reason I am asking is that I create multiple workspaces for p4 in jenkinsfile, in this case one for android and one for ios, so jenkins workspace should look like this

            MyRoot
                ios/p4sync...
                android/p4sync...
            MyRoot@script/p4sync/JenkinsfileAndStuff
            MyRoot@tmp

            but instead

            MyRoot
                 p4sync/JenkinsfileAndStuff
                 ios/p4sync...
                 android/p4sync...
            MyRoot@script/p4sync/JenkinsfileAndStuff
            MyRoot@tmp

            So it looks like I have one extra p4sync.

            To make things little more annoying we are using pollScm as a trigger right now so our view mapping basically has to have full project and not only Jenkinsfile so we have 2 location with full sync that are not used for the actual build. 

             

            ynedelin yury nedelin added a comment - Hi Paul, thanks for your help, everything has been working fairly well for us in terms of getting the job done. I wanted to double check one thing, with pipeline method will p4 create two locations with full sync indicated by view mapping ? I see full set of files in MyRoot@script and also in MyRoot, is that how it should work? The reason I am asking is that I create multiple workspaces for p4 in jenkinsfile, in this case one for android and one for ios, so jenkins workspace should look like this MyRoot     ios/p4sync...     android/p4sync... MyRoot@script/p4sync/JenkinsfileAndStuff MyRoot@tmp but instead MyRoot      p4sync / JenkinsfileAndStuff      ios/p4sync...      android/p4sync... MyRoot@script/p4sync / JenkinsfileAndStuff MyRoot@tmp So it looks like I have one extra p4sync. To make things little more annoying we are using pollScm as a trigger right now so our view mapping basically has to have full project and not only Jenkinsfile so we have 2 location with full sync that are not used for the actual build.   

            re: https://issues.jenkins-ci.org/browse/JENKINS-45999?focusedCommentId=310943&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-310943 , specifically "I would recommend limiting the View so only the script (or scripts if required)  are checked out and not all your source and assets."

            How would this be applied when it comes to MultiBranch Pipeline jobs? I've got a stream depot with a checked in Jenkinsfile in each stream, which I of course can add whatever checkout step neccessary for the actual build, but I don't understand how I can still have Jenkins discover new streams and create subsequent jobs. Am I missing something?

            Best,

            D

            danielresolutiongames Daniel Hagström added a comment - re: https://issues.jenkins-ci.org/browse/JENKINS-45999?focusedCommentId=310943&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-310943  , specifically "I would recommend limiting the View so only the script (or scripts if required)  are checked out and not all your source and assets." How would this be applied when it comes to MultiBranch Pipeline jobs? I've got a stream depot with a checked in Jenkinsfile in each stream, which I of course can add whatever checkout step neccessary for the actual build, but I don't understand how I can still have Jenkins discover new streams and create subsequent jobs. Am I missing something? Best, D
            anttifm Antti Hirvonen added a comment - - edited

            I would be also very interested in knowing how to do a lightweight checkout on master node with streams. We have big repositories with lots of binary assets so doing a full checkout on master node is wasteful in terms of build time and disk utilization. We are also utilizing multibranch build system so this is especially bad for projects which have many active branches.

            If there's a fix in progress, can you give any ETA when it would ship? Any way to circumvent this for now?

            Other than that I have no complaints and I'm so happy we are finally able to utilize full power of Jenkins with Perforce. Keep up the good work!

            Thanks, Antti

            anttifm Antti Hirvonen added a comment - - edited I would be also very interested in knowing how to do a lightweight checkout on master node with streams. We have big repositories with lots of binary assets so doing a full checkout on master node is wasteful in terms of build time and disk utilization. We are also utilizing multibranch build system so this is especially bad for projects which have many active branches. If there's a fix in progress, can you give any ETA when it would ship? Any way to circumvent this for now? Other than that I have no complaints and I'm so happy we are finally able to utilize full power of Jenkins with Perforce. Keep up the good work! Thanks, Antti
            p4paul Paul Allen added a comment -

            Hi Guys, I am looking to implement SCMFileSystem and related classes:

            https://github.com/jenkinsci/scm-api-plugin/blob/master/src/main/java/jenkins/scm/api/SCMFileSystem.java

            This will allow Jenkins to browse/navigate Perforce and find the individual files it needs.  I may need to do additional work to resolve the specific problems in this and other issues, but it looks like the SCMFileSystem is the Jenkins way, although Perforce Views have been an alternative in the interim.   I'm adding LightWeightCheckout to the roadmap, but doubt I'll have anything ready before the end of Jan, beginning of Feb.

            For Multibranch streams, you could create a child stream which only shares the Jenkinsfile and associated scripts; unfortunately Virtual streams don't work in this situation, so you will need a full 'development' stream. 

            p4paul Paul Allen added a comment - Hi Guys, I am looking to implement SCMFileSystem and related classes: https://github.com/jenkinsci/scm-api-plugin/blob/master/src/main/java/jenkins/scm/api/SCMFileSystem.java This will allow Jenkins to browse/navigate Perforce and find the individual files it needs.  I may need to do additional work to resolve the specific problems in this and other issues, but it looks like the SCMFileSystem is the Jenkins way, although Perforce Views have been an alternative in the interim.   I'm adding LightWeightCheckout to the roadmap, but doubt I'll have anything ready before the end of Jan, beginning of Feb. For Multibranch streams, you could create a child stream which only shares the Jenkinsfile and associated scripts; unfortunately Virtual streams don't work in this situation, so you will need a full 'development' stream. 

            Thanks for the update. We'll be waiting for the fix.

            anttifm Antti Hirvonen added a comment - Thanks for the update. We'll be waiting for the fix.

            Code changed in jenkins
            User: Paul Allen
            Path:
            src/main/java/org/jenkinsci/plugins/p4/client/NavigateHelper.java
            src/main/java/org/jenkinsci/plugins/p4/client/TempClientHelper.java
            src/main/java/org/jenkinsci/plugins/p4/scm/AbstractP4ScmSource.java
            src/main/java/org/jenkinsci/plugins/p4/scm/BranchesScmSource.java
            src/main/java/org/jenkinsci/plugins/p4/scm/P4SCMFile.java
            src/main/java/org/jenkinsci/plugins/p4/scm/P4SCMFileSystem.java
            src/test/java/org/jenkinsci/plugins/p4/scm/P4SCMFileSystemTest.java
            http://jenkins-ci.org/commit/p4-plugin/3ef0aa574af14b320d0fd45847ca3ceb0f6fc3a4
            Log:
            Lightweight checkout support.

            Implementation for scm-api classes SCMFile and SCMFileSystem, allowing
            Jenkins to navigate Perforce within the scope of a workspace view.

            Lightweight checkout uses a tempoary Perforce workspace to naviagete
            and fetch the files. The client name and client view mapping will be
            modified from a template name e.g. jenkins-${NODE_NAME}-${JOB_NAME} to
            the tempoary name jenkinsTemp-UUID. Alternativly if a user as used
            ${P4_CLIENT} in the client mapping this will remain unchanged and will
            be get expanded during the job run.

            JENKINS-45999
            JENKINS-46269

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Paul Allen Path: src/main/java/org/jenkinsci/plugins/p4/client/NavigateHelper.java src/main/java/org/jenkinsci/plugins/p4/client/TempClientHelper.java src/main/java/org/jenkinsci/plugins/p4/scm/AbstractP4ScmSource.java src/main/java/org/jenkinsci/plugins/p4/scm/BranchesScmSource.java src/main/java/org/jenkinsci/plugins/p4/scm/P4SCMFile.java src/main/java/org/jenkinsci/plugins/p4/scm/P4SCMFileSystem.java src/test/java/org/jenkinsci/plugins/p4/scm/P4SCMFileSystemTest.java http://jenkins-ci.org/commit/p4-plugin/3ef0aa574af14b320d0fd45847ca3ceb0f6fc3a4 Log: Lightweight checkout support. Implementation for scm-api classes SCMFile and SCMFileSystem, allowing Jenkins to navigate Perforce within the scope of a workspace view. Lightweight checkout uses a tempoary Perforce workspace to naviagete and fetch the files. The client name and client view mapping will be modified from a template name e.g. jenkins-${NODE_NAME}-${JOB_NAME} to the tempoary name jenkinsTemp-UUID. Alternativly if a user as used ${P4_CLIENT} in the client mapping this will remain unchanged and will be get expanded during the job run. JENKINS-45999 JENKINS-46269
            p4paul Paul Allen added a comment -

            Ready for release.

            Please try build 146

            p4paul Paul Allen added a comment - Ready for release. Please try build 146

            Excellent, will definitely try this as soon as possible!

            anttifm Antti Hirvonen added a comment - Excellent, will definitely try this as soon as possible!
            p4paul Paul Allen added a comment -

            Released in 1.8.4

            p4paul Paul Allen added a comment - Released in 1.8.4

            I'm trying to understand what this means: 

            The client name and client view mapping will be
            modified from a template name e.g. jenkins-${NODE_NAME}-${JOB_NAME} to
            the temporary name jenkinsTemp-UUID. Alternativly if a user as used
            ${P4_CLIENT} in the client mapping this will remain unchanged and will
            be get expanded during the job run.

            I would like it to mean that if I specify my own "Workspace name" and then use "${P4_CLIENT}" in "View Mappings", then it will use my workspace name instead of jenkinsTemp-UUID. That doesn't appear to work this way. View Mapping remains unchanged but "Workspace name gets replaced.

            This is annoying as it is creating (and not destroying) a large number of perforce workspaces which is bogging down our p4 server.

            Is this supposed to allow user override of the workspace name, if so then I believe that it doesn't do that (on version 1.8.10)?

            rg Russell Gallop added a comment - I'm trying to understand what this means:  The client name and client view mapping will be modified from a template name e.g. jenkins-${NODE_NAME}-${JOB_NAME} to the temporary name jenkinsTemp-UUID. Alternativly if a user as used ${P4_CLIENT} in the client mapping this will remain unchanged and will be get expanded during the job run. I would like it to mean that if I specify my own "Workspace name" and then use "${P4_CLIENT}" in "View Mappings", then it will use my workspace name instead of jenkinsTemp-UUID. That doesn't appear to work this way. View Mapping remains unchanged but "Workspace name gets replaced. This is annoying as it is creating (and not destroying) a large number of perforce workspaces which is bogging down our p4 server. Is this supposed to allow user override of the workspace name, if so then I believe that it doesn't do that (on version 1.8.10)?
            p4paul Paul Allen added a comment -

            Hi Russel - I have open an new issue to track the changes to the lightweight checkout.

            JENKINS-54488

            p4paul Paul Allen added a comment - Hi Russel - I have open an new issue to track the changes to the lightweight checkout. JENKINS-54488

            People

              p4paul Paul Allen
              karthik546 karthik paidi
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: