Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47978

Lightweight checkout not working for branches that contain forward slash '/'

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • Jenkins version 2.73.3
      bitbucket-branch-source-plugin version 2.2.7

      I am using bitbucket-branch-source-plugin version 2.2.7 (currently the latest one) and lightweight checkout for Jenkinsfile fails for branches that contain forward slash '/'. I tried with different branches like ft/*. 

      The functionality fallbacks to heavyweight checkout, so it keeps working. Here is the exception that I get:

      ERROR: Could not do lightweight checkout, falling back to heavyweight
      java.io.FileNotFoundException: URL: https://api.bitbucket.org/2.0/repositories/owner/repository/src/ft%2Fci/Jenkinsfile
      	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequestAsInputStream(BitbucketCloudApiClient.java:558)
      	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getFileContent(BitbucketCloudApiClient.java:719)
      	at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:81)
      	at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:263)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:421)

       

      The functionality works perfectly for branches like master, dev, development and release.

          [JENKINS-47978] Lightweight checkout not working for branches that contain forward slash '/'

          Vivek Pandey added a comment -

          Fix released in bitbucket-branch-source 2.2.9.

          Vivek Pandey added a comment - Fix released in bitbucket-branch-source 2.2.9.

          bjorn rohlen added a comment - - edited

          This fix seriously broke something. We've been at it for a week – it still will not find Jenkinsfile in subdirectories. 

          Started by user P*** S**** (p*.**)
          [Thu Jan 25 10:00:01 CET 2018] Starting branch indexing...
          Connecting to http://**.*.*.se using allowed.user/******Repository type: Git
          Looking up asd/asd_stuff for branches
          Checking branch feature/asd-things from asd/asd_stuff
          ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found
          Skipped
          Checking branch feature/ASD-123-test-doit from asd/asd_stuff
          ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found
          Skipped
          ...

          ... although the file is clearly present and worked before this patch. Jenkinsfile in root path of a repo will work just fine, in a subdirectory, not so fine. 

          To clarify: If you specify Jenkinsfile in a subdirectory, that will no longer work. I've reproduced this numerous times before re-opening this issue on my own repos. 

          bjorn rohlen added a comment - - edited This fix seriously broke something. We've been at it for a week – it still will not find Jenkinsfile in subdirectories.  Started by user P*** S**** (p* . **) [Thu Jan 25 10:00:01 CET 2018] Starting branch indexing... Connecting to http://** . * . *.se using allowed.user/******Repository type: Git Looking up asd/asd_stuff for branches Checking branch feature/asd-things from asd/asd_stuff ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found Skipped Checking branch feature/ASD-123-test-doit from asd/asd_stuff ‘javaproject/asd-stuff-autotest/Jenkinsfile’ not found Skipped ... ... although the file is clearly present and worked before this patch. Jenkinsfile in root path of a repo will work just fine, in a subdirectory, not so fine.  To clarify: If you specify Jenkinsfile in a subdirectory, that will no longer work. I've reproduced this numerous times before re-opening this issue on my own repos. 

          bjorn rohlen added a comment - - edited

          REOPENING ISSUE

          .. broke other things containing slash '/', see previous comment.

          bjorn rohlen added a comment - - edited REOPENING ISSUE .. broke other things containing slash '/', see previous comment.

          Vivek Pandey added a comment -

          bjro Ok, so to confirm Jenkinsfile is getting picked correctly when at its located at repository/branch root. If placed inside a subdirectory its not detected.

          I checked with github branch source as well, Jenkinsfile discovery doesn't get picked from subdirectory. Looks like this is consistent compared to github branch source.

          stephenconnolly can you shade some light as whats expected behavior here? 

          Vivek Pandey added a comment - bjro  Ok, so to confirm Jenkinsfile is getting picked correctly when at its located at repository/branch root. If placed inside a subdirectory its not detected. I checked with github branch source as well, Jenkinsfile discovery doesn't get picked from subdirectory. Looks like this is consistent compared to github branch source. stephenconnolly can you shade some light as whats expected behavior here? 

          If the SCMSourceCriteria is looking for a file at a non-root path, that is supposed to work. If it is not working for GitHub then that is a bug in GitHub, as is not working for Bitbucket here.

          https://github.com/jenkinsci/workflow-multibranch-plugin/pull/59 was the PR that enabled the use case in multibranch pipelines and is aligned with the architectural designs of SCM API.

          Stephen Connolly added a comment - If the SCMSourceCriteria is looking for a file at a non-root path, that is supposed to work. If it is not working for GitHub then that is a bug in GitHub, as is not working for Bitbucket here. https://github.com/jenkinsci/workflow-multibranch-plugin/pull/59  was the PR that enabled the use case in multibranch pipelines and is aligned with the architectural designs of SCM API.

          it is possible that GitHub Branch Source has a regression, but https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMFileSystemTest.java#L193 is passing (yes based on an evil replayed capture of real github), so unless https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMProbe.java#L135-L151 has a bug, we should be detecting the Jenkinsfile at the path and creating the branch, and then loading the file from the lightweight checkout (which the test "claims" is working)

           

          So it looks like GitHub is working with child paths... of course I don't trust the wiremock tests and would love to rip them out... but GitHub has that rate limit that makes testing live a nightmare!

          Stephen Connolly added a comment - it is possible that GitHub Branch Source has a regression, but https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMFileSystemTest.java#L193  is passing (yes based on an evil replayed capture of real github), so unless https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMProbe.java#L135-L151  has a bug, we should be detecting the Jenkinsfile at the path and creating the branch, and then loading the file from the lightweight checkout (which the test "claims" is working)   So it looks like GitHub is working with child paths... of course I don't trust the wiremock tests and would love to rip them out... but GitHub has that rate limit that makes testing live a nightmare!

          bjorn rohlen added a comment -

          Well, unfortunately it does not, as confirmed by Vivek and myself.

          This is still an issue and breaking projects. 

          bjorn rohlen added a comment - Well, unfortunately it does not, as confirmed by Vivek and myself. This is still an issue and breaking projects. 

          Anna Tikhonova added a comment - - edited

          I can confirm that the original issue is still there. I don't know if it is in Bitbucket branch source plugin or elsewhere. I have tested 2.2.10 (which is the latest at the time of writing) with Bitbucket Server and Jenkinsfile placed in build/test.jenkinsfile in my repository. With 'master' branch it is flawless. However, when I put it to branch 'feature/test', in the console output I have:

          Lightweight checkout support not available, falling back to full checkout.
          Checking out git ssh://localhost:7999/bitbucket/~atikhonova/test.git into /var/lib/jenkins/workspace/test@script to read build/test.jenkinsfile
          

          Anna Tikhonova added a comment - - edited I can confirm that the original issue is still there. I don't know if it is in Bitbucket branch source plugin or elsewhere. I have tested 2.2.10 (which is the latest at the time of writing) with Bitbucket Server and Jenkinsfile placed in build/test.jenkinsfile in my repository. With 'master' branch it is flawless. However, when I put it to branch 'feature/test', in the console output I have: Lightweight checkout support not available, falling back to full checkout. Checking out git ssh: //localhost:7999/bitbucket/~atikhonova/test.git into / var /lib/jenkins/workspace/test@script to read build/test.jenkinsfile

          Anna Tikhonova added a comment - - edited

          The workaround is to put 'refs/heads/feature/test' in Branch Specifier field of Git SCM. Not 'feature/test', not '*/feature/test'. You have to specify your git ref at full as this is the only supported case. See the breaking code in git-plugin (src/main/java//jenkins/plugins/git/GitSCMFileSystem.java):

          @Extension(ordinal = Short.MIN_VALUE)
          public static class BuilderImpl extends SCMFileSystem.Builder {
          
            @Override
            public boolean supports(SCM source) {
              return source instanceof GitSCM
                && ((GitSCM) source).getUserRemoteConfigs().size() == 1
                && ((GitSCM) source).getBranches().size() == 1
                && ((GitSCM) source).getBranches().get(0).getName().matches(
                "^((\\Q" + Constants.R_HEADS + "\\E.*)|([^/]+)|(\\*/[^/*]+(/[^/*]+)*))$"        // HERE !!
              );
            // we only support where the branch spec is obvious

          I think this simple implementation is good and the only issue here is the match failing silently. Probably, this method should put a warning to the Console explaining why lightweight checkout failed.

          Anna Tikhonova added a comment - - edited The workaround is to put 'refs/heads/feature/test' in Branch Specifier field of Git SCM. Not 'feature/test', not '*/feature/test'. You have to specify your git ref at full as this is the only supported case. See the breaking code in git-plugin (src/main/java//jenkins/plugins/git/GitSCMFileSystem.java): @Extension(ordinal = Short .MIN_VALUE) public static class BuilderImpl extends SCMFileSystem.Builder { @Override public boolean supports(SCM source) { return source instanceof GitSCM && ((GitSCM) source).getUserRemoteConfigs().size() == 1   && ((GitSCM) source).getBranches().size() == 1   && ((GitSCM) source).getBranches().get(0).getName().matches( "^((\\Q" + Constants.R_HEADS + "\\E.*)|([^/]+)|(\\*/[^/*]+(/[^/*]+)*))$" // HERE !! ); // we only support where the branch spec is obvious I think this simple implementation is good and the only issue here is the match failing silently. Probably, this method should put a warning to the Console explaining why lightweight checkout failed.

          Recently tested it with 2.2.13 at least it seems fixed.

          Joseph Petersen (old) added a comment - Recently tested it with 2.2.13 at least it seems fixed.

            jetersen Joseph Petersen
            ghristov88 Georgi Hristov
            Votes:
            6 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: