Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47978

Lightweight checkout not working for branches that contain forward slash '/'

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • Jenkins version 2.73.3
      bitbucket-branch-source-plugin version 2.2.7

    Description

      I am using bitbucket-branch-source-plugin version 2.2.7 (currently the latest one) and lightweight checkout for Jenkinsfile fails for branches that contain forward slash '/'. I tried with different branches like ft/*. 

      The functionality fallbacks to heavyweight checkout, so it keeps working. Here is the exception that I get:

      ERROR: Could not do lightweight checkout, falling back to heavyweight
      java.io.FileNotFoundException: URL: https://api.bitbucket.org/2.0/repositories/owner/repository/src/ft%2Fci/Jenkinsfile
      	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequestAsInputStream(BitbucketCloudApiClient.java:558)
      	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getFileContent(BitbucketCloudApiClient.java:719)
      	at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:81)
      	at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:263)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:421)

       

      The functionality works perfectly for branches like master, dev, development and release.

      Attachments

        Issue Links

          Activity

            stephanbudge Stephan Budge added a comment -

            This is also affecting me.

            stephanbudge Stephan Budge added a comment - This is also affecting me.
            olympusmonds Luke Mondy added a comment -

            I've had this same issue, I've put up a pull-request to hopefully fix it:

            https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/84

            olympusmonds Luke Mondy added a comment - I've had this same issue, I've put up a pull-request to hopefully fix it: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/84

            abayer, this is a similar problem, but not exactly the same. I think that the one that you are referring to ( JENKINS-47911 ) was fixed in 2.2.7.

            In this version (2.2.7) the escaping of the '/' character is fixed and the exception is not thrown, but the escaped branch name can't be found by the Bitbucket API and the functionality fallbacks to the heavyweight checkout.

            ghristov88 Georgi Hristov added a comment - abayer , this is a similar problem, but not exactly the same. I think that the one that you are referring to ( JENKINS-47911 ) was fixed in 2.2.7. In this version (2.2.7) the escaping of the '/' character is fixed and the exception is not thrown, but the escaped branch name can't be found by the Bitbucket API and the functionality fallbacks to the heavyweight checkout.
            olympusmonds Luke Mondy added a comment -

            ghristov88 I found 2.2.7 still had the '/' not being escaped, hence my PR - but I think you are also right about the branch name not working.

            olympusmonds Luke Mondy added a comment - ghristov88 I found 2.2.7 still had the '/' not being escaped, hence my PR - but I think you are also right about the branch name not working.

            Wasn't the complete fix

            stephenconnolly Stephen Connolly added a comment - Wasn't the complete fix

            2.2.8

            stephenconnolly Stephen Connolly added a comment - 2.2.8

            Still failing in 2.2.8, with BitBucket Cloud and both Git and Mercurial

            luispiedra Luis Piedra-Márquez added a comment - Still failing in 2.2.8, with BitBucket Cloud and both Git and Mercurial
            vivek Vivek Pandey added a comment -

            Bitbucket Cloud 2.0 API to get source (file/directory) content should be in context of revision hash and not branch name.

            https://api.bitbucket.org/2.0/repositories/owner/repository/src/{node}/{path}
            

            I checked with BB cloud API team, the node parameter could be simple branch name but not all. Safe to use hash to have it work in all cases.

            Opened PR https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/102

            vivek Vivek Pandey added a comment - Bitbucket Cloud 2.0 API to get source (file/directory) content should be in context of revision hash and not branch name. https: //api.bitbucket.org/2.0/repositories/owner/repository/src/{node}/{path} I checked with BB cloud API team, the node parameter could be simple branch name but not all. Safe to use hash to have it work in all cases. Opened PR https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/102
            vivek Vivek Pandey added a comment -

            Fix released in bitbucket-branch-source 2.2.9.

            vivek Vivek Pandey added a comment - Fix released in bitbucket-branch-source 2.2.9.
            bjro bjorn rohlen added a comment - - edited

            This fix seriously broke something. We've been at it for a week – it still will not find Jenkinsfile in subdirectories. 

            Started by user P*** S**** (p*.**)
            [Thu Jan 25 10:00:01 CET 2018] Starting branch indexing...
            Connecting to http://**.*.*.se using allowed.user/******Repository type: Git
            Looking up asd/asd_stuff for branches
            Checking branch feature/asd-things from asd/asd_stuff
            ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found
            Skipped
            Checking branch feature/ASD-123-test-doit from asd/asd_stuff
            ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found
            Skipped
            ...

            ... although the file is clearly present and worked before this patch. Jenkinsfile in root path of a repo will work just fine, in a subdirectory, not so fine. 

            To clarify: If you specify Jenkinsfile in a subdirectory, that will no longer work. I've reproduced this numerous times before re-opening this issue on my own repos. 

            bjro bjorn rohlen added a comment - - edited This fix seriously broke something. We've been at it for a week – it still will not find Jenkinsfile in subdirectories.  Started by user P*** S**** (p* . **) [Thu Jan 25 10:00:01 CET 2018] Starting branch indexing... Connecting to http://** . * . *.se using allowed.user/******Repository type: Git Looking up asd/asd_stuff for branches Checking branch feature/asd-things from asd/asd_stuff ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found Skipped Checking branch feature/ASD-123-test-doit from asd/asd_stuff ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found Skipped ... ... although the file is clearly present and worked before this patch. Jenkinsfile in root path of a repo will work just fine, in a subdirectory, not so fine.  To clarify: If you specify Jenkinsfile in a subdirectory, that will no longer work. I've reproduced this numerous times before re-opening this issue on my own repos. 
            bjro bjorn rohlen added a comment - - edited

            REOPENING ISSUE

            .. broke other things containing slash '/', see previous comment.

            bjro bjorn rohlen added a comment - - edited REOPENING ISSUE .. broke other things containing slash '/', see previous comment.
            vivek Vivek Pandey added a comment -

            bjro Ok, so to confirm Jenkinsfile is getting picked correctly when at its located at repository/branch root. If placed inside a subdirectory its not detected.

            I checked with github branch source as well, Jenkinsfile discovery doesn't get picked from subdirectory. Looks like this is consistent compared to github branch source.

            stephenconnolly can you shade some light as whats expected behavior here? 

            vivek Vivek Pandey added a comment - bjro  Ok, so to confirm Jenkinsfile is getting picked correctly when at its located at repository/branch root. If placed inside a subdirectory its not detected. I checked with github branch source as well, Jenkinsfile discovery doesn't get picked from subdirectory. Looks like this is consistent compared to github branch source. stephenconnolly can you shade some light as whats expected behavior here? 

            If the SCMSourceCriteria is looking for a file at a non-root path, that is supposed to work. If it is not working for GitHub then that is a bug in GitHub, as is not working for Bitbucket here.

            https://github.com/jenkinsci/workflow-multibranch-plugin/pull/59 was the PR that enabled the use case in multibranch pipelines and is aligned with the architectural designs of SCM API.

            stephenconnolly Stephen Connolly added a comment - If the SCMSourceCriteria is looking for a file at a non-root path, that is supposed to work. If it is not working for GitHub then that is a bug in GitHub, as is not working for Bitbucket here. https://github.com/jenkinsci/workflow-multibranch-plugin/pull/59  was the PR that enabled the use case in multibranch pipelines and is aligned with the architectural designs of SCM API.

            it is possible that GitHub Branch Source has a regression, but https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMFileSystemTest.java#L193 is passing (yes based on an evil replayed capture of real github), so unless https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMProbe.java#L135-L151 has a bug, we should be detecting the Jenkinsfile at the path and creating the branch, and then loading the file from the lightweight checkout (which the test "claims" is working)

             

            So it looks like GitHub is working with child paths... of course I don't trust the wiremock tests and would love to rip them out... but GitHub has that rate limit that makes testing live a nightmare!

            stephenconnolly Stephen Connolly added a comment - it is possible that GitHub Branch Source has a regression, but https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMFileSystemTest.java#L193  is passing (yes based on an evil replayed capture of real github), so unless https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMProbe.java#L135-L151  has a bug, we should be detecting the Jenkinsfile at the path and creating the branch, and then loading the file from the lightweight checkout (which the test "claims" is working)   So it looks like GitHub is working with child paths... of course I don't trust the wiremock tests and would love to rip them out... but GitHub has that rate limit that makes testing live a nightmare!
            bjro bjorn rohlen added a comment -

            Well, unfortunately it does not, as confirmed by Vivek and myself.

            This is still an issue and breaking projects. 

            bjro bjorn rohlen added a comment - Well, unfortunately it does not, as confirmed by Vivek and myself. This is still an issue and breaking projects. 
            atikhono Anna Tikhonova added a comment - - edited

            I can confirm that the original issue is still there. I don't know if it is in Bitbucket branch source plugin or elsewhere. I have tested 2.2.10 (which is the latest at the time of writing) with Bitbucket Server and Jenkinsfile placed in build/test.jenkinsfile in my repository. With 'master' branch it is flawless. However, when I put it to branch 'feature/test', in the console output I have:

            Lightweight checkout support not available, falling back to full checkout.
            Checking out git ssh://localhost:7999/bitbucket/~atikhonova/test.git into /var/lib/jenkins/workspace/test@script to read build/test.jenkinsfile
            
            atikhono Anna Tikhonova added a comment - - edited I can confirm that the original issue is still there. I don't know if it is in Bitbucket branch source plugin or elsewhere. I have tested 2.2.10 (which is the latest at the time of writing) with Bitbucket Server and Jenkinsfile placed in build/test.jenkinsfile in my repository. With 'master' branch it is flawless. However, when I put it to branch 'feature/test', in the console output I have: Lightweight checkout support not available, falling back to full checkout. Checking out git ssh: //localhost:7999/bitbucket/~atikhonova/test.git into / var /lib/jenkins/workspace/test@script to read build/test.jenkinsfile
            atikhono Anna Tikhonova added a comment - - edited

            The workaround is to put 'refs/heads/feature/test' in Branch Specifier field of Git SCM. Not 'feature/test', not '*/feature/test'. You have to specify your git ref at full as this is the only supported case. See the breaking code in git-plugin (src/main/java//jenkins/plugins/git/GitSCMFileSystem.java):

            @Extension(ordinal = Short.MIN_VALUE)
            public static class BuilderImpl extends SCMFileSystem.Builder {
            
              @Override
              public boolean supports(SCM source) {
                return source instanceof GitSCM
                  && ((GitSCM) source).getUserRemoteConfigs().size() == 1
                  && ((GitSCM) source).getBranches().size() == 1
                  && ((GitSCM) source).getBranches().get(0).getName().matches(
                  "^((\\Q" + Constants.R_HEADS + "\\E.*)|([^/]+)|(\\*/[^/*]+(/[^/*]+)*))$"        // HERE !!
                );
              // we only support where the branch spec is obvious

            I think this simple implementation is good and the only issue here is the match failing silently. Probably, this method should put a warning to the Console explaining why lightweight checkout failed.

            atikhono Anna Tikhonova added a comment - - edited The workaround is to put 'refs/heads/feature/test' in Branch Specifier field of Git SCM. Not 'feature/test', not '*/feature/test'. You have to specify your git ref at full as this is the only supported case. See the breaking code in git-plugin (src/main/java//jenkins/plugins/git/GitSCMFileSystem.java): @Extension(ordinal = Short .MIN_VALUE) public static class BuilderImpl extends SCMFileSystem.Builder { @Override public boolean supports(SCM source) { return source instanceof GitSCM && ((GitSCM) source).getUserRemoteConfigs().size() == 1   && ((GitSCM) source).getBranches().size() == 1   && ((GitSCM) source).getBranches().get(0).getName().matches( "^((\\Q" + Constants.R_HEADS + "\\E.*)|([^/]+)|(\\*/[^/*]+(/[^/*]+)*))$" // HERE !! ); // we only support where the branch spec is obvious I think this simple implementation is good and the only issue here is the match failing silently. Probably, this method should put a warning to the Console explaining why lightweight checkout failed.

            Recently tested it with 2.2.13 at least it seems fixed.

            casz Joseph Petersen (old) added a comment - Recently tested it with 2.2.13 at least it seems fixed.

            People

              jetersen Joseph Petersen
              ghristov88 Georgi Hristov
              Votes:
              6 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: