Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48420

Provide lightweight checkout capability for Subversion

    XMLWordPrintable

Details

    • 2.12.0

    Description

      JENKINS-33273 introduced a lightweight checkout capability to download a Jenkinsfile from source control without checkout out the entire repository, but it appears this hasn't been implemented for Subversion: "Lightweight checkout support not available, falling back to full checkout."

      Attachments

        Issue Links

          Activity

            I have a working fork on GitHub with this functionality, what it the best way to get the community to validate it? Should I assign the JIRA to me and issue a PR, or something else?

            andre4s_m Andreas Magnusson added a comment - I have a working fork on GitHub with this functionality, what it the best way to get the community to validate it? Should I assign the JIRA to me and issue a PR, or something else?
            oleg_nenashev Oleg Nenashev added a comment -

            andre4s_m yes, it would be the best approach

            oleg_nenashev Oleg Nenashev added a comment - andre4s_m yes, it would be the best approach

            andre4s_m: any news on this? I'd be interested, too. (Link to the hinted GitHub would also be a good start)

            ilmarinen Mathias Helminger added a comment - andre4s_m : any news on this? I'd be interested, too. (Link to the hinted GitHub would also be a good start)

            ilmarinen: Life interfered, but here is the GitHub project: https://github.com/Zune01/subversion-plugin/branches

            andre4s_m Andreas Magnusson added a comment - ilmarinen : Life interfered, but here is the GitHub project: https://github.com/Zune01/subversion-plugin/branches

            andre4s_m: Thanks for the link to the GitHub project. Did you send the PR?

            jmalopoy Jesús Malo Poyatos added a comment - andre4s_m : Thanks for the link to the GitHub project. Did you send the PR?

            I am currently trying to migrate some of my organization's existing jobs to use Jenkinsfiles instead. We'd like to store the Jenkinsfiles in some of our SVN paths since new branches are regularly created which need jobs created automatically, so managing jobs with a multibranch pipeline job scanning for Jenkinsfiles is a great solution for us, but I am currently running into this SVN limitation.

            Although we allow plenty of space on our agents for checkouts, our repos are large enough that we risk running out of space on our master Jenkins machine if a fully recursive checkout is done there simply to read the Jenkinsfile. Will this feature be available relatively soon?

            tflanigan Thomas Flanigan added a comment - I am currently trying to migrate some of my organization's existing jobs to use Jenkinsfiles instead. We'd like to store the Jenkinsfiles in some of our SVN paths since new branches are regularly created which need jobs created automatically, so managing jobs with a multibranch pipeline job scanning for Jenkinsfiles is a great solution for us, but I am currently running into this SVN limitation. Although we allow plenty of space on our agents for checkouts, our repos are large enough that we risk running out of space on our master Jenkins machine if a fully recursive checkout is done there simply to read the Jenkinsfile. Will this feature be available relatively soon?

            This commit contains the fixes for lightweight checkouts, right? https://github.com/Zune01/subversion-plugin/commit/100d8fbefb3d68733b26439daf94cded6cd9fbb8

            (We really need this fix as we are going to run out of inodes)

            elmuerte Michiel Hendriks added a comment - This commit contains the fixes for lightweight checkouts, right? https://github.com/Zune01/subversion-plugin/commit/100d8fbefb3d68733b26439daf94cded6cd9fbb8 (We really need this fix as we are going to run out of inodes)

            I've tested the changes against the current plugin versions and it looks like everything works correctly. I've issued a pull request for these changes: https://github.com/jenkinsci/subversion-plugin/pull/221

            elmuerte Michiel Hendriks added a comment - I've tested the changes against the current plugin versions and it looks like everything works correctly. I've issued a pull request for these changes: https://github.com/jenkinsci/subversion-plugin/pull/221

            Looks like one of the added test cases fails on the buildserver (work for me locally): SubversionSCMFileSystemTest#lastModified_Smokes

            Not sure what the added value of this test is as it mostly relies on svnkit's "lastModified" response. But there is clearly a timing issue going on. i.e. buildserver not being fast enough for this test to fall within the given time window.

             

            elmuerte Michiel Hendriks added a comment - Looks like one of the added test cases fails on the buildserver (work for me locally): SubversionSCMFileSystemTest#lastModified_Smokes Not sure what the added value of this test is as it mostly relies on svnkit's "lastModified" response. But there is clearly a timing issue going on. i.e. buildserver not being fast enough for this test to fall within the given time window.  

            Kevin Cathcart created a new pull request with fixed unit test: https://github.com/jenkinsci/subversion-plugin/pull/223

            It passes all tests. (So I've closed the #221 PR in favor of the #223)

            elmuerte Michiel Hendriks added a comment - Kevin Cathcart created a new pull request with fixed unit test: https://github.com/jenkinsci/subversion-plugin/pull/223 It passes all tests. (So I've closed the #221 PR in favor of the #223)
            ziggystar Thomasero Vulturera added a comment - - edited

            I just did an update to the new version. It breaks my jobs in a Pipeline-Multibranch job because the sparse checkout apparently cannot follow svn:externals. There is also no option within the Pipeline-Multibranch plugin to disable sparse checkout, so I had to downgrade the plugin.

            I have 50+ permanent branches, and I want all of them to use the same Jenkinsfile. So I put the script into a separate folder and import that folder into my branch folder and have a symlink into the external:

            shared/
              Jenkinsfile
            <branch>/ (with shared/ as svn:external under shared/)
              Jenkinsfile -> shared/Jenkinsfile

            I could get rid of this setup with file externals (don't remember why this didn't work). But still, the Subversion plugin in sparse mode needs to follow externals for my usecase to work.

             

            ziggystar Thomasero Vulturera added a comment - - edited I just did an update to the new version. It breaks my jobs in a Pipeline-Multibranch job because the sparse checkout apparently cannot follow svn:externals. There is also no option within the Pipeline-Multibranch plugin to disable sparse checkout, so I had to downgrade the plugin. I have 50+ permanent branches, and I want all of them to use the same Jenkinsfile. So I put the script into a separate folder and import that folder into my branch folder and have a symlink into the external: shared/   Jenkinsfile <branch>/ (with shared/ as svn:external under shared/)   Jenkinsfile -> shared/Jenkinsfile I could get rid of this setup with file externals (don't remember why this didn't work). But still, the Subversion plugin in sparse mode needs to follow externals for my usecase to work.  

            I think it could have easy workaround with a property here https://github.com/KevinCathcart/subversion-plugin/blob/d70e95d6cfe2ea660449417fa54abdeec51b203e/src/main/java/jenkins/scm/impl/subversion/SubversionSCMFileSystem.java#L74-L82 I will make the PR and release a minor version with the workaround

            ifernandezcalvo Ivan Fernandez Calvo added a comment - I think it could have easy workaround with a property here https://github.com/KevinCathcart/subversion-plugin/blob/d70e95d6cfe2ea660449417fa54abdeec51b203e/src/main/java/jenkins/scm/impl/subversion/SubversionSCMFileSystem.java#L74-L82  I will make the PR and release a minor version with the workaround
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            I just releases 2.12.1, it contains a new property to disable lightweight checkout -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=<true/false> for SCM API 2.0 so it would use the default implementation.

            By default the SCM API 2.0 is used (the new implementation here), set the property means to use the implementation on <2.12.0 versions

            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited I just releases 2.12.1, it contains a new property to disable lightweight checkout -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=<true/false> for SCM API 2.0 so it would use the default implementation. By default the SCM API 2.0 is used (the new implementation here), set the property means to use the implementation on <2.12.0 versions
            psteiner Philip Steiner added a comment - - edited

            After upgrading SVNPlugin from v2.11.1 to v2.12.1, pipeline jobs on my Jenkins v2.107.3 master started throwing an SVN exception (see example below).

            The pipeline jobs are defined as "Pipeline script from SCM" with "Lightweight Checkout" selected (default when SVN is added to job).

            Check-out Strategy is "Always check out a fresh copy". Quiet checkout is selected.

            If I deselect "Lightweight Checkout" in the job configuration, then the pipeline checkout step succeeds.

            org.tmatesoft.svn.core.SVNException: svn: E160013: '/svn/gpd/!svn/bc/59394/CI/dev/trunk/CI/GPD60/AE/jenkinsfile' path not found: 404 Not Found (https://svnyvr1.mda.ca)
            {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)}}
            {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:920)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doGet(DAVConnection.java:335)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:289)}}
            {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:251)}}
            {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:82)}}
            Caused: java.io.IOException: failed to fetch file: CI/GPD60/AE/jenkinsfile
            {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:85)}}
            {{ at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)}}
            {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:110)}}
            {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)}}
            {{ at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)}}
            {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}}
            {{ at hudson.model.Executor.run(Executor.java:429)}}
            Finished: FAILURE

             

            psteiner Philip Steiner added a comment - - edited After upgrading SVNPlugin from v2.11.1 to v2.12.1, pipeline jobs on my Jenkins v2.107.3 master started throwing an SVN exception (see example below). The pipeline jobs are defined as "Pipeline script from SCM" with "Lightweight Checkout" selected (default when SVN is added to job). Check-out Strategy is "Always check out a fresh copy". Quiet checkout is selected. If I deselect "Lightweight Checkout" in the job configuration, then the pipeline checkout step succeeds. org.tmatesoft.svn.core.SVNException: svn: E160013: '/svn/gpd/!svn/bc/59394/CI/dev/trunk/CI/GPD60/AE/jenkinsfile' path not found: 404 Not Found ( https://svnyvr1.mda.ca ) {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:70)}} {{ at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:57)}} {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:760)}} {{ at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:920)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doGet(DAVConnection.java:335)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:289)}} {{ at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getFile(DAVRepository.java:251)}} {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:82)}} Caused: java.io.IOException: failed to fetch file: CI/GPD60/AE/jenkinsfile {{ at jenkins.scm.impl.subversion.SubversionSCMFile.content(SubversionSCMFile.java:85)}} {{ at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)}} {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:110)}} {{ at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)}} {{ at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)}} {{ at hudson.model.ResourceController.execute(ResourceController.java:97)}} {{ at hudson.model.Executor.run(Executor.java:429)}} Finished: FAILURE  
            jpraet Jimmy Praet added a comment -

            Since upgrading to v2.12.x, our jenkins pipeline builds always showed "No changes".

            I configured -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true as a workaround.

            jpraet Jimmy Praet added a comment - Since upgrading to v2.12.x, our jenkins pipeline builds always showed "No changes". I configured -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true as a workaround.

            psteiner Is the jenkinsfile in an external? if so you will need to set -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true property.

            ifernandezcalvo Ivan Fernandez Calvo added a comment - psteiner Is the jenkinsfile in an external? if so you will need to set -Djenkins.scm.impl.subversion.SubversionSCMFileSystem.disable=true property.

            ifernandezcalvo - I don't think there are any externals. The files all reside in the same subdirectory of the repository, and as far as I know, there are no external references (I am not an SVN expert).

            I have worked around the error for now by deselecting "Lightweight checkout" on all jobs that use these pipeline files.

            Forgive this question, I am just an end-user on this Jenkins master, with no admin access - where would I ask the sysadmin to set the property you suggested? Is that in a configuration file for the Jenkins master?

            psteiner Philip Steiner added a comment - ifernandezcalvo - I don't think there are any externals. The files all reside in the same subdirectory of the repository, and as far as I know, there are no external references (I am not an SVN expert). I have worked around the error for now by deselecting "Lightweight checkout" on all jobs that use these pipeline files. Forgive this question, I am just an end-user on this Jenkins master, with no admin access - where would I ask the sysadmin to set the property you suggested? Is that in a configuration file for the Jenkins master?

            psteiner you have detailed steps to make it on this KB

            ifernandezcalvo Ivan Fernandez Calvo added a comment - psteiner you have detailed steps to make it on this KB
            psteiner Philip Steiner added a comment - - edited

            Thanks ifernandezcalvo I will forward the KB to my sys admin

            psteiner Philip Steiner added a comment - - edited Thanks ifernandezcalvo I will forward the KB to my sys admin
            rockowwc Robert Orrock added a comment -

            psteiner did the java argument fix your issue?

            rockowwc Robert Orrock added a comment - psteiner did the java argument fix your issue?

            rockowwc Yes it did! 

            psteiner Philip Steiner added a comment - rockowwc Yes it did! 

            I'm having the same issue than Philip Steiner. I'm sure I don't have externals. What is the fix ?

            richard_a Arnaud Richard added a comment - I'm having the same issue than Philip Steiner. I'm sure I don't have externals. What is the fix ?

            People

              ifernandezcalvo Ivan Fernandez Calvo
              mgrochowalski Matt Grochowalski
              Votes:
              14 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: