Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42518

Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

    • Blue Ocean 1.4 - beta 2

      JENKINS-33273 introduced a lightweight checkout capability to allow SCM implementors to avoid having checkout the scm on the master to read the contents of the Jenkinsfile from the SCM.  The comments in that bug mention that git doesn't offer a way to do so but I would think this is possible to do using Bitbucket APIs to read a particular file's contents on a branch.  

      In our use case, our repository is multi-gigabyte and we have many developers working within it creating many branches so cloning the repo on master for every branch is very costly in terms of time and storage.

          [JENKINS-42518] Provide lightweight checkout capability for bitbucket to avoid repository clone for multi-branch pipeline jobs

          Georgi Hristov added a comment - - edited

          I think there is a problem when trying to lightweight checkout of Jenkins file for branches that have '/' in them like (ft/*)

          ERROR: Could not do lightweight checkout, falling back to heavyweight
          java.io.FileNotFoundException: URL: https://api.bitbucket.org/2.0/repositories/owner/repository/src/ft%2Fci/Jenkinsfile
          	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequestAsInputStream(BitbucketCloudApiClient.java:558)
          	at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getFileContent(BitbucketCloudApiClient.java:719)
          	at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:81)
          	at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338)
          	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104)
          	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:263)
          	at hudson.model.ResourceController.execute(ResourceController.java:97)
          	at hudson.model.Executor.run(Executor.java:421)
          

           

          For branches like master, dev, release it works perfectly.

          Thanks

           

          Georgi Hristov added a comment - - edited I think there is a problem when trying to lightweight checkout of Jenkins file for branches that have '/' in them like (ft/*) ERROR: Could not do lightweight checkout, falling back to heavyweight java.io.FileNotFoundException: URL: https: //api.bitbucket.org/2.0/repositories/owner/repository/src/ft%2Fci/Jenkinsfile at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequestAsInputStream(BitbucketCloudApiClient.java:558) at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getFileContent(BitbucketCloudApiClient.java:719) at com.cloudbees.jenkins.plugins.bitbucket.filesystem.BitbucketSCMFile.content(BitbucketSCMFile.java:81) at jenkins.scm.api.SCMFile.contentAsString(SCMFile.java:338) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:263) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421)   For branches like master, dev, release it works perfectly. Thanks  

          Michael Neale added a comment -

          ghristov88 it was a problem with versions early than 2.2.7 - are you running the latest? 

          Michael Neale added a comment - ghristov88 it was a problem with versions early than 2.2.7 - are you running the latest? 

          Michael Neale added a comment -

          kgiloo I expect in some cases with PRs it may need to clone it to do a validated merge (github plugin has this same challenge). Woudl that be right vivek ?

          Michael Neale added a comment - kgiloo I expect in some cases with PRs it may need to clone it to do a validated merge (github plugin has this same challenge). Woudl that be right vivek ?

          michaelneale yes I am running 2.2.7. What can I provide to help you reproduce the issue?

          Georgi Hristov added a comment - michaelneale yes I am running 2.2.7. What can I provide to help you reproduce the issue?

          Michael Neale added a comment - - edited

          ghristov88 perhaps should put this in a new issue, as it looks a bit distinct from what others are seeing. 

          Just reproduction instructions should be doable, I expect it is more encoding confusion where it can't find the resource, so falls back to what it knows works, but I assume when it falls back it does work at least. 

          Michael Neale added a comment - - edited ghristov88 perhaps should put this in a new issue, as it looks a bit distinct from what others are seeing.  Just reproduction instructions should be doable, I expect it is more encoding confusion where it can't find the resource, so falls back to what it knows works, but I assume when it falls back it does work at least. 

          michaelneale, yes it falls back and it works. I will open a new issue later today.

          Georgi Hristov added a comment - michaelneale , yes it falls back and it works. I will open a new issue later today.

          I am also seeing the same issue as Georgi using 2.2.7 plugin version.  I'm running with Bitbucket Server version 4.11.  Is there a Bitbucket version requirement for the API used to be available?

          NetAppBlueDevil added a comment - I am also seeing the same issue as Georgi using 2.2.7 plugin version.  I'm running with Bitbucket Server version 4.11.  Is there a Bitbucket version requirement for the API used to be available?

          Is this feature toggle-able? Upgraded today from 2.2.3 and it randomly broke pipelines across the board. All of our jobs were configured to use the stash step in the @script directory to take advantage of that initial checkout. After upgrading to 2.2.7, jobs will sometimes do the heavy checkout while others will sometimes do the light checkout, with no real distinction as to why.

          Do we now need to explicitly have a step in our pipeline code to checkout out the repository? I've rolled back to 2.2.3 but I don't want to keep relying on an old version of the plugin. Does anyone have a recommendation on how to proceed?

          Aaron Lunsford added a comment - Is this feature toggle-able? Upgraded today from 2.2.3 and it randomly broke pipelines across the board. All of our jobs were configured to use the stash step in the @script directory to take advantage of that initial checkout. After upgrading to 2.2.7, jobs will sometimes do the heavy checkout while others will sometimes do the light checkout, with no real distinction as to why. Do we now need to explicitly have a step in our pipeline code to checkout out the repository? I've rolled back to 2.2.3 but I don't want to keep relying on an old version of the plugin. Does anyone have a recommendation on how to proceed?

          Wolff added a comment - - edited

          Is it possible that Atlassian changed some licencing needed for this? We are running into the following error:

          com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl handleErrorBitbucket - did not accept the request
          Failed to retrieve mirroring information for project Test and repo Testing
          com.atlassian.bitbucket.jenkins.internal.client.exception.BadRequestException: - response: 409 with body: '{"errors":[{"context":null,"message":"Mirroring requires a Bitbucket Data Center license.","exceptionName":"com.atlassian.bitbucket.mirroring.upstream.MirroringDisabledException"}]}'
          

          Unfortunately Jenkins output does not specify which request is not accepted, but since we also get

          Lightweight checkout support not available, falling back to full checkout.
          

          I assume the first part is the cause.

          Wolff added a comment - - edited Is it possible that Atlassian changed some licencing needed for this? We are running into the following error: com.atlassian.bitbucket.jenkins.internal.http.HttpRequestExecutorImpl handleErrorBitbucket - did not accept the request Failed to retrieve mirroring information for project Test and repo Testing com.atlassian.bitbucket.jenkins.internal.client.exception.BadRequestException: - response: 409 with body: '{ "errors" :[{ "context" : null , "message" : "Mirroring requires a Bitbucket Data Center license." , "exceptionName" : "com.atlassian.bitbucket.mirroring.upstream.MirroringDisabledException" }]}' Unfortunately Jenkins output does not specify which request is not accepted, but since we also get Lightweight checkout support not available, falling back to full checkout. I assume the first part is the cause.

          Kalle Niemitalo added a comment - - edited

          wlfbck, the package name com.atlassian.bitbucket.jenkins.internal.http belongs to atlassian-bitbucket-server-integration-plugin, but this issue is for bitbucket-branch-source-plugin. I suggest you file a separate issue on atlassian-bitbucket-server-integration-plugin.

          The "Lightweight checkout support not available" message for atlassian-bitbucket-server-integration-plugin is being addressed in JENKINS-63033.

          Kalle Niemitalo added a comment - - edited wlfbck , the package name com.atlassian.bitbucket.jenkins.internal.http belongs to atlassian-bitbucket-server-integration-plugin, but this issue is for bitbucket-branch-source-plugin. I suggest you file a separate issue on atlassian-bitbucket-server-integration-plugin. The "Lightweight checkout support not available" message for atlassian-bitbucket-server-integration-plugin is being addressed in JENKINS-63033 .

            vivek Vivek Pandey
            petehayes Peter Hayes
            Votes:
            18 Vote for this issue
            Watchers:
            33 Start watching this issue

              Created:
              Updated:
              Resolved: