Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-75231

Bitbucket fails to checkout shared library using SSH after upgrade to 934.4.1

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major

      After upgrading Bitbucket Branch Source plugin from 934.3.2 to 934.4.1, pipelines are unable to checkout shared libraries (using ssh username with private key credentials).

      Rolling back to 934.3.2 fixes the problem.

      Relevant pipeline execution log:

      Obtained Jenkinsfile from 94d0e48ab7d02974ad07acb2f648c7661d418ab6
      Loading library shared-libs@main
      Attempting to resolve main from remote references... 
      > git --version # timeout=10 
      > git --version # timeout=10
      using GIT_SSH to set credentials my-ssh-creds
      Verifying host key using known hosts file
      ERROR: Checkout failed
      java.io.IOException: error=2, No such file or directory
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
      Caused: java.io.IOException: Cannot run program "cp": error=2, No such file or directory
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createNonBusyExecutable(CliGitAPIImpl.java:2745)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createUnixGitSSH(CliGitAPIImpl.java:2726)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2120)
      Caused: hudson.plugins.git.GitException: Failed to setup credentials
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2186)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2080)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2071)
        at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:3806)
        at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:885)
      Caused: java.io.IOException
        at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:887)
        at PluginClassLoader for scm-api//jenkins.scm.api.SCMSource.fetch(SCMSource.java:635)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:75)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.SCMBasedRetriever.retrySCMOperation(SCMBasedRetriever.java:222)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:75)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:298)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:157)
        at PluginClassLoader for pipeline-groovy-lib//org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
        at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:652)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:598)
        at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:445)
      ERROR: Maximum checkout retry attempts reached, aborting
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:WorkflowScript: Loading libraries failed
      
      1 error
        at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309) 
        at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1107)
        at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624) 
        at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
        at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
        at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
        at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
        at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:652)
        at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:598)
        at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
        at hudson.model.ResourceController.execute(ResourceController.java:101)
        at hudson.model.Executor.run(Executor.java:445)
      

          [JENKINS-75231] Bitbucket fails to checkout shared library using SSH after upgrade to 934.4.1

          Nikolas Falco added a comment -

          It's bitbucket server or cloud?
          If server are you using a mirror server?

          Nikolas Falco added a comment - It's bitbucket server or cloud? If server are you using a mirror server?

          Pierre-Julien added a comment -

          I'm sorry, I should've mentioned it in the environment, we are using Bitbucket Cloud.

          Pierre-Julien added a comment - I'm sorry, I should've mentioned it in the environment, we are using Bitbucket Cloud.

          Nikolas Falco added a comment - - edited

          I have read better the first row of the issue:

          pipelines are unable to checkout shared libraries

          We never get success to download libraries using the SSH key, for this reason we forced to register an app password.

          Could you post the library configuration are you using? Mask the repo and owner, I would replicate the issue in my local PC (both 934.3.2 and master revision)

          Nikolas Falco added a comment - - edited I have read better the first row of the issue: pipelines are unable to checkout shared libraries We never get success to download libraries using the SSH key, for this reason we forced to register an app password. Could you post the library configuration are you using? Mask the repo and owner, I would replicate the issue in my local PC (both 934.3.2 and master revision)

          Pierre-Julien added a comment -

          I added a screenshot of the configs I'm using for this shared libraries. This has worked flawlessly with SSH checkout until version 934.4.1, rolling back this plugin made it work again (nothing else was required to fix it).

          Pierre-Julien added a comment - I added a screenshot of the configs I'm using for this shared libraries. This has worked flawlessly with SSH checkout until version 934.4.1, rolling back this plugin made it work again (nothing else was required to fix it).

          Nikolas Falco added a comment -

          From the screenshot ssh credential seems to be valid because it find the "current revision"

          I had configured the same on my windows PC and it works. Did you cut some row from console log?
          This is my log:

          Avviato dall'utente unknown or anonymous
          Obtained Jenkinsfile from 8e64830465435886f59d5512d9de7af969ccfd40
          Loading library devops@test
          Attempting to resolve test from remote references...
           > git.exe --version # timeout=10
           > git --version # 'git version 2.47.1.windows.2'
          using GIT_SSH to set credentials ssh
          Verifying host key using known hosts file
           > git.exe ls-remote -h -- git@bitbucket.org:acme/***-pipelines.git # timeout=10
          Found match: refs/heads/test revision 3f0bde1a9bca5ff05b59df8c6165d055070bce65
          The recommended git tool is: NONE
          using credential ssh
          Cloning the remote Git repository
          Cloning with configured refspecs honoured and without tags
          Cloning repository git@bitbucket.org:acme/***-pipelines.git
           > git.exe init D:\git\bitbucket-branch-source-plugin\work\workspace\backend_feature_test@libs\14a5668c8112612833f8336ac9a165c0493af2011aa84d5350ea355ab3afa29b # timeout=10
          Fetching upstream changes from git@bitbucket.org:acme/***-pipelines.git
           > git.exe --version # timeout=10
           > git --version # 'git version 2.47.1.windows.2'
          using GIT_SSH to set credentials ssh
          Verifying host key using known hosts file
           > git.exe fetch --no-tags --force --progress -- git@bitbucket.org:acme/***-pipelines.git +refs/heads/*:refs/remotes/origin/* # timeout=10
           > git.exe config remote.origin.url git@bitbucket.org:acme/***-pipelines.git # timeout=10
           > git.exe config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
          Avoid second fetch
          Checking out Revision 3f0bde1a9bca5ff05b59df8c6165d055070bce65 (test)
           > git.exe config core.sparsecheckout # timeout=10
           > git.exe checkout -f 3f0bde1a9bca5ff05b59df8c6165d055070bce65 # timeout=10
          Commit message: "test"
          

          Nikolas Falco added a comment - From the screenshot ssh credential seems to be valid because it find the "current revision" I had configured the same on my windows PC and it works. Did you cut some row from console log? This is my log: Avviato dall'utente unknown or anonymous Obtained Jenkinsfile from 8e64830465435886f59d5512d9de7af969ccfd40 Loading library devops@test Attempting to resolve test from remote references... > git.exe --version # timeout=10 > git --version # 'git version 2.47.1.windows.2' using GIT_SSH to set credentials ssh Verifying host key using known hosts file > git.exe ls-remote -h -- git@bitbucket.org:acme/***-pipelines.git # timeout=10 Found match: refs/heads/test revision 3f0bde1a9bca5ff05b59df8c6165d055070bce65 The recommended git tool is: NONE using credential ssh Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository git@bitbucket.org:acme/***-pipelines.git > git.exe init D:\git\bitbucket-branch-source-plugin\work\workspace\backend_feature_test@libs\14a5668c8112612833f8336ac9a165c0493af2011aa84d5350ea355ab3afa29b # timeout=10 Fetching upstream changes from git@bitbucket.org:acme/***-pipelines.git > git.exe --version # timeout=10 > git --version # 'git version 2.47.1.windows.2' using GIT_SSH to set credentials ssh Verifying host key using known hosts file > git.exe fetch --no-tags --force --progress -- git@bitbucket.org:acme/***-pipelines.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git.exe config remote.origin.url git@bitbucket.org:acme/***-pipelines.git # timeout=10 > git.exe config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 3f0bde1a9bca5ff05b59df8c6165d055070bce65 (test) > git.exe config core.sparsecheckout # timeout=10 > git.exe checkout -f 3f0bde1a9bca5ff05b59df8c6165d055070bce65 # timeout=10 Commit message: "test"

          Pierre-Julien added a comment -

          I did cut the bottom of the log, but it was not related at all to the problem, the rest is exactly as it is. I do think, though, from what I see in the log, that the problem is not with the authentication, but when trying to exec 'cp' in the checkout routine. I can't say if that has anything to do with my setup or not...

          Pierre-Julien added a comment - I did cut the bottom of the log, but it was not related at all to the problem, the rest is exactly as it is. I do think, though, from what I see in the log, that the problem is not with the authentication, but when trying to exec 'cp' in the checkout routine. I can't say if that has anything to do with my setup or not...

          We're running into the same issue. Rollback to 934.3.2 fixed it for us as well.

          We fetch the shared library using BitBucket OAuth authentication. I've attached the log:failing-pipeline.txt

          Hildebrand Soethoudt added a comment - We're running into the same issue. Rollback to 934.3.2 fixed it for us as well. We fetch the shared library using BitBucket OAuth authentication. I've attached the log: failing-pipeline.txt

            nfalco Nikolas Falco
            pjhoude Pierre-Julien
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: