Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73797

Git checkout of Bitbucket multibranch Pipelines fails when using ssh

      Since release 5.5.0 of the Git-Client the "security" → "Git Host Key Verification Configuration" setting is ignored (at least on Kubernetes Agents).

      Workaround is manual downgrade to 5.4.1

      Using anything other than "Known hosts files" causes "stderr: Host key verification failed." followed by "fatal: Could not read from remote repository."

      Checking out the pipeline repos beforehand is still working, seems to be a problem on kubernetes agents builds only.

        1. image-2024-09-20-13-37-05-669.png
          image-2024-09-20-13-37-05-669.png
          37 kB
        2. plugins.txt
          2 kB
        3. run-jenkins.sh
          0.9 kB
        4. sysinfo.txt
          156 kB

          [JENKINS-73797] Git checkout of Bitbucket multibranch Pipelines fails when using ssh

          Same here. Multibranch bitbucket pipeline seems only affected. Not regular ones

          Valentin Delaye added a comment - Same here. Multibranch bitbucket pipeline seems only affected. Not regular ones

          Ruby Paasche added a comment - - edited

          yoerg markewaite

          Looks like the BitBucket multibranch plugin overrides the
          auth and with "reordering of extension additions" the Creds from the Bitbucket multibranch plugin gets active and not the one from the git plugin.

          Ruby Paasche added a comment - - edited yoerg markewaite Looks like the BitBucket multibranch plugin overrides the auth and with "reordering of extension additions" the Creds from the Bitbucket multibranch plugin gets active and not the one from the git plugin.

          Ruby Paasche added a comment -

          Tried it:

          PR-1657 Host key verification failed.
          PR-1658 Verifying host key using manually-configured host key entries

          Can confirm this.

          Ruby Paasche added a comment - Tried it: PR-1657 Host key verification failed. PR-1658 Verifying host key using manually-configured host key entries Can confirm this.

          Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Valentin Delaye added a comment - Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Daniel added a comment - - edited

          I found another workaround beside downgrading the plugin. 

          1.) 
          In case you use the Bitbucket multibranch setup, do not checkout via ssh anymore, use the PAT token instead. 

          With the ssh checkout, we had the effect that all pull request jobs directly fail. Non pull request jobs were somehow not affected. Without the ssh checkout, the pull request jobs start again. 

           

          2.) In case that you invoke the checkout step directly in your Jenkinsfile, consider the following.
               Try to explicitly inject the credentials instead of relying on the scm variable.
               So, instead of:

              gitSCM = checkout(
                [
                  $class                           : 'GitSCM',
                  branches                         : scm.branches,
                  doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
                  userRemoteConfigs                : scm.userRemoteConfigs,
                  extensions                       : [
                    [$class   : "CloneOption",
                     noTags   : noTags,
                     shallow  : isShallowClone,
                     depth    : cloneDepth,
                     reference: ''],
                  ],
                ],
              )

          Use something like:

              gitSCM = checkout(
                [
                  $class                           : 'GitSCM',
                  branches                         : scm.branches,
                  doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
                  userRemoteConfigs                : [[url: scm.userRemoteConfigs[0].url, credentialsId: "YourCredentialsID",
                  extensions                       : [
                    [$class   : "CloneOption",
                     noTags   : noTags,
                     shallow  : isShallowClone,
                     depth    : cloneDepth,
                     reference: ''],
                  ],
                ],
              )

           

          Maybe this is helpful for somebody.

          Daniel added a comment - - edited I found another workaround beside downgrading the plugin.  1.)  In case you use the Bitbucket multibranch setup, do not checkout via ssh anymore, use the PAT token instead.  With the ssh checkout, we had the effect that all pull request jobs directly fail. Non pull request jobs were somehow not affected. Without the ssh checkout, the pull request jobs start again.    2.) In case that you invoke the checkout step directly in your Jenkinsfile, consider the following.      Try to explicitly inject the credentials instead of relying on the scm variable.      So, instead of:     gitSCM = checkout(       [         $class                           : 'GitSCM',         branches                         : scm.branches,         doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,         userRemoteConfigs                : scm.userRemoteConfigs,         extensions                       : [           [$class   : "CloneOption",            noTags   : noTags,            shallow  : isShallowClone,            depth    : cloneDepth,            reference: ''],         ],       ],     ) Use something like:     gitSCM = checkout(       [         $class                           : 'GitSCM',         branches                         : scm.branches,         doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,         userRemoteConfigs                : [[url: scm.userRemoteConfigs [0] .url, credentialsId: "YourCredentialsID",         extensions                       : [           [$class   : "CloneOption",            noTags   : noTags,            shallow  : isShallowClone,            depth    : cloneDepth,            reference: ''],         ],       ],     )   M aybe this is helpful for somebody.

          Mark Waite added a comment -

          Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Thanks for the suggestion. I'm releasing git plugin 5.5.1 that removes the pull request so that I then have time to implement the change with appropriate test environment improvements that will detect this type of failure. I believe that my test environment already has the ability to show this failure, but I didn't check the status of every test job after making that change. I knew the change was risky and had evaluated it in several different ways (my environment, plugin tests, plugin BOM, acceptance test harness), but did not specifically check the behavior of Bitbucket branch source plugin multibranch Pipelines with ssh authentication. I believe that I have those job types, but I did not check their status.

          Mark Waite added a comment - Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ? Thanks for the suggestion. I'm releasing git plugin 5.5.1 that removes the pull request so that I then have time to implement the change with appropriate test environment improvements that will detect this type of failure. I believe that my test environment already has the ability to show this failure, but I didn't check the status of every test job after making that change. I knew the change was risky and had evaluated it in several different ways (my environment, plugin tests, plugin BOM, acceptance test harness), but did not specifically check the behavior of Bitbucket branch source plugin multibranch Pipelines with ssh authentication. I believe that I have those job types, but I did not check their status.

          Mark Waite added a comment -

          I've released git plugin 5.5.1 that reverts the reordering of extension additions. My sincere apologies to everyone affected by the issue.

          I'll work on additional automation that detects that type of failure during pull request evaluation, since the additional automation will help many other pull requests as well.

          Mark Waite added a comment - I've released git plugin 5.5.1 that reverts the reordering of extension additions. My sincere apologies to everyone affected by the issue. I'll work on additional automation that detects that type of failure during pull request evaluation, since the additional automation will help many other pull requests as well.

          Pay Bas added a comment -

          My sincere apologies to everyone affected by the issue.

          I don't see why you should apologize. This is an open-source project maintained largely by volunteers and we're all using it for free.

          Besides: the ecosystem is massive, so every possible plugin interaction is impossible to test beforehand.

          We should be thanking you for your dedication and rapid response.

          Pay Bas added a comment - My sincere apologies to everyone affected by the issue. I don't see why you should apologize. This is an open-source project maintained largely by volunteers and we're all using it for free. Besides: the ecosystem is massive, so every possible plugin interaction is impossible to test beforehand. We should be thanking you for your dedication and rapid response.

          It's not just host key verification, actually testing this the GitSCM was using the Pipeline Multibranch scan credentials instead of the SSH Credentials (that could check in the build log that is not showing using GIT_SSH anymore. Bitbucket Branch source in the scenario of SSHCheckoutTrait configures a GitSCM with the SSH credentials but also an authenticator extension (supposed to decorate the GitClient with inferred authentication from the Pipeline Multibranch scan credentials). Which is probably wrong.
          The way authentication is pre-configures is something I was trying to address as part of https://issues.jenkins.io/browse/JENKINS-73471. I have added a test for this scenario in Bitbucket Branch Source to cover this https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/867/files. Hopefully this covers it for the next attempt to apply JENKINS-73677

          Allan BURDAJEWICZ added a comment - It's not just host key verification, actually testing this the GitSCM was using the Pipeline Multibranch scan credentials instead of the SSH Credentials (that could check in the build log that is not showing using GIT_SSH anymore. Bitbucket Branch source in the scenario of SSHCheckoutTrait configures a GitSCM with the SSH credentials but also an authenticator extension (supposed to decorate the GitClient with inferred authentication from the Pipeline Multibranch scan credentials). Which is probably wrong. The way authentication is pre-configures is something I was trying to address as part of https://issues.jenkins.io/browse/JENKINS-73471 . I have added a test for this scenario in Bitbucket Branch Source to cover this https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/867/files . Hopefully this covers it for the next attempt to apply JENKINS-73677

          Should further be prevented by bitbucket-branch-source-plugin:906.vedf430cb_4481.

          Allan BURDAJEWICZ added a comment - Should further be prevented by bitbucket-branch-source-plugin:906.vedf430cb_4481.

            markewaite Mark Waite
            rpaasche Ruby Paasche
            Votes:
            6 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: