Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73797

Git checkout of Bitbucket multibranch Pipelines fails when using ssh

      Since release 5.5.0 of the Git-Client the "security" → "Git Host Key Verification Configuration" setting is ignored (at least on Kubernetes Agents).

      Workaround is manual downgrade to 5.4.1

      Using anything other than "Known hosts files" causes "stderr: Host key verification failed." followed by "fatal: Could not read from remote repository."

      Checking out the pipeline repos beforehand is still working, seems to be a problem on kubernetes agents builds only.

        1. sysinfo.txt
          156 kB
        2. run-jenkins.sh
          0.9 kB
        3. plugins.txt
          2 kB
        4. image-2024-09-20-13-37-05-669.png
          image-2024-09-20-13-37-05-669.png
          37 kB

          [JENKINS-73797] Git checkout of Bitbucket multibranch Pipelines fails when using ssh

          Yoerg added a comment -

          Same here, with OpenStack cloud agent and "Manually provided keys" strategy. Regression in 5.5.0, downgrade to 5.4.1 fixes the issue.

          Yoerg added a comment - Same here, with OpenStack cloud agent and "Manually provided keys" strategy. Regression in 5.5.0, downgrade to 5.4.1 fixes the issue.

          Mark Waite added a comment -

          Thanks for reporting the issue rpaasche.

          Can you confirm that when you upgraded to git plugin 5.5.0 that you also upgraded git client plugin? The git client plugin handles the host key verification configuration. There were host key verification changes in git client plugin 5.0.0 If you were running git client plugin 4.7.0 before and upgraded to git client plugin 5.0.0, that would narrow the search range for my investigation.

          You mention that it is specific to kubernetes agents. Can you share the other agent types that you have tried?

          Can you share the list of installed plugins before the problem and the list of installed plugins after the problem appeared? "How to report an issue" includes a script that generates the list of installed plugins on a running system.

          Mark Waite added a comment - Thanks for reporting the issue rpaasche . Can you confirm that when you upgraded to git plugin 5.5.0 that you also upgraded git client plugin? The git client plugin handles the host key verification configuration. There were host key verification changes in git client plugin 5.0.0 If you were running git client plugin 4.7.0 before and upgraded to git client plugin 5.0.0, that would narrow the search range for my investigation. You mention that it is specific to kubernetes agents. Can you share the other agent types that you have tried? Can you share the list of installed plugins before the problem and the list of installed plugins after the problem appeared? "How to report an issue" includes a script that generates the list of installed plugins on a running system.

          Mark Waite added a comment -

          Thanks yoerg. Same questions apply to your comment. Can you provide the detailed list of plugins installed before the problem was seen and after the problem was seen?

          Mark Waite added a comment - Thanks yoerg . Same questions apply to your comment. Can you provide the detailed list of plugins installed before the problem was seen and after the problem was seen?

          Yoerg added a comment - - edited

          Right, sorry: Jenkins LTS 2.462.2 on Java 21.0.4+7-Ubuntu-1ubuntu222.04

          I didn't change anything besides the Git plugin.

          Working configuration:

          Failing configuration:

          • Git client plugin 5.0.0
          • Git plugin 5.5.0

          Git client plugin 6 requires Jenkins 2.463 so it's not available for the current LTS

          Yoerg added a comment - - edited Right, sorry: Jenkins LTS 2.462.2 on Java 21.0.4+7-Ubuntu-1ubuntu222.04 I didn't change anything besides the Git plugin . Working configuration: sysinfo.txt Git client plugin 5.0.0 Git plugin 5.4.1 Failing configuration: Git client plugin 5.0.0 Git plugin 5.5.0 Git client plugin 6 requires Jenkins 2.463 so it's not available for the current LTS

          Pay Bas added a comment - - edited

          Can confirm. Also had to downgrade to 5.4.1 due to:

          ERROR: Error cloning remote repo 'origin'
            hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- ssh://git@bitbucket.mega.corp:7999/project/repo.git +refs/heads/task/PROJ-83195-my-branch:refs/remotes/origin/task/PROJ-83195-my-branch +refs/heads/develop:refs/remotes/origin/develop" returned status code 128:
            stdout: 
            stderr: git@bitbucket.mega.corp: Permission denied (publickey).
            fatal: Could not read from remote repository.

          Pay Bas added a comment - - edited Can confirm. Also had to downgrade to 5.4.1 due to: ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- ssh: //git@bitbucket.mega.corp:7999/project/repo.git +refs/heads/task/PROJ-83195-my-branch:refs/remotes/origin/task/PROJ-83195-my-branch +refs/heads/develop:refs/remotes/origin/develop" returned status code 128: stdout: stderr: git@bitbucket.mega.corp: Permission denied (publickey). fatal: Could not read from remote repository.

          Karl added a comment -

          I am having the same problem ... agent is a AWS cloud agent, not k8s.

          • Git client plugin 5.0.0
          • Git plugin 5.5.0
          • Jenkins 2.462.2 (LTS) on Ubuntu

          Karl added a comment - I am having the same problem ... agent is a AWS cloud agent, not k8s. Git client plugin 5.0.0 Git plugin 5.5.0 Jenkins 2.462.2 (LTS) on Ubuntu

          Ruby Paasche added a comment - - edited

          markewaite

          Git-Client 5.0.0 + Git 5.5.0 ⇾ doesn't work
          Git-Client 5.0.0 + Git 5.4.1 ⇾ works
          Git-Client 6.0.0 + Git 5.4.1 ⇾ causes `java.lang.NullPointerException: Cannot invoke "jenkins.scm.api.SCMHead.getName()" because "this.head" is null` (but was expected, we are on the LTS version)
          Git-Client 6.0.0 + Git 5.5.0 ⇾ doesn't work

          Ruby Paasche added a comment - - edited markewaite Git-Client 5.0.0 + Git 5.5.0 ⇾ doesn't work Git-Client 5.0.0 + Git 5.4.1 ⇾ works Git-Client 6.0.0 + Git 5.4.1 ⇾ causes `java.lang.NullPointerException: Cannot invoke "jenkins.scm.api.SCMHead.getName()" because "this.head" is null` (but was expected, we are on the LTS version) Git-Client 6.0.0 + Git 5.5.0 ⇾ doesn't work

          Mark Waite added a comment - - edited

          Thanks for the additional reports. That further confirms there is a real problem, though I don't yet know the source of the problem.

          I'm not able to duplicate the problem in my configuration testing. I'm using a static agent in various forms rather than using a Kubernetes agent. I would really appreciate additional details that would allow me to duplicate it, especially if it can be duplicated without requiring that I install and configure a Kubernetes cluster.

          Steps that I've attempted while trying to duplicate the problem:

          1. Create a plugins.txt file that defines the plugins to be used
          2. Create a run-jenkins.sh script that downloads Jenkins 2.462.2 and the specified plugin versions
          3. Run the run-jenkins.sh script and complete the setup wizard by creating a user and choosing to install no additional plugins
          4. Configure the controller with 0 executors and define an inbound agent
          5. Create a new user account on a Linux computer and connect the inbound agent with that account (so that it has no values for known hosts)
          6. Set the host key configuration to use manually configured host keys and add the host key for GitHub.com
          7. Define a Jenkins private key credential that is known to my user account on GitHub.com
          8. Define a Jenkins job that clones from a private GitHub.com repository and uses the define private key credential
          9. Run the job and confirm that it is successful
          10. Remove the host key for GitHub.com from the manually configured host keys on the Jenkins controller
          11. Run the job and confirm that it fails because the host key is not available

          Mark Waite added a comment - - edited Thanks for the additional reports. That further confirms there is a real problem, though I don't yet know the source of the problem. I'm not able to duplicate the problem in my configuration testing. I'm using a static agent in various forms rather than using a Kubernetes agent. I would really appreciate additional details that would allow me to duplicate it, especially if it can be duplicated without requiring that I install and configure a Kubernetes cluster. Steps that I've attempted while trying to duplicate the problem: Create a plugins.txt file that defines the plugins to be used Create a run-jenkins.sh script that downloads Jenkins 2.462.2 and the specified plugin versions Run the run-jenkins.sh script and complete the setup wizard by creating a user and choosing to install no additional plugins Configure the controller with 0 executors and define an inbound agent Create a new user account on a Linux computer and connect the inbound agent with that account (so that it has no values for known hosts) Set the host key configuration to use manually configured host keys and add the host key for GitHub.com Define a Jenkins private key credential that is known to my user account on GitHub.com Define a Jenkins job that clones from a private GitHub.com repository and uses the define private key credential Run the job and confirm that it is successful Remove the host key for GitHub.com from the manually configured host keys on the Jenkins controller Run the job and confirm that it fails because the host key is not available

          Mark Waite added a comment -

          Additional testing with an Alpine agent from the docker plugin using an Alpine does not show the issue either. I confirmed that manually provided host keys are sent to the Alpine agent that is started by the docker plugin. The Alpine agent that I'm using is the most recent release https://hub.docker.com/r/jenkins/inbound-agent/tags?page_size=&ordering=&name=3261.v9c670a_4748a_9-4-alpine

          Mark Waite added a comment - Additional testing with an Alpine agent from the docker plugin using an Alpine does not show the issue either. I confirmed that manually provided host keys are sent to the Alpine agent that is started by the docker plugin. The Alpine agent that I'm using is the most recent release https://hub.docker.com/r/jenkins/inbound-agent/tags?page_size=&ordering=&name=3261.v9c670a_4748a_9-4-alpine

          Pay Bas added a comment - - edited

          We are using our own built agents, which (extremely simplified) boils down to:

          • fedora:40 base image
          • Oracle JDK 17.0.12
          • remoting jar 3261.v9c670a_4748a_9

          In combination with Jenkins LTS 2.462.2 on Fedora 40 + java-11-openjdk-11.0.24.0.8-2.fc40.x86_64

          The infrastructure the agent runs on doesn't seem to make a difference. I'm getting the issues both when running our agents through the Jenkins Kubernetes plugin (on OpenShift) as well as simply running the agent as a traditional Docker container on a regular VM. Which leads me to believe this has nothing to do with Kubernetes.

          Our init script (again extremely simplified):

          JAVA_OPTS="-Dhudson.slaves.WorkspaceList='_'"
          JNLP_JAVA_OPTIONS="-Xms512m -Xmx512m -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90" 
          
          exec ${JAVA17_HOME}/bin/java ${JAVA_OPTS} ${JNLP_JAVA_OPTIONS} -jar /usr/share/jenkins/agent.jar -url ${JENKINS_URL} -name ${JENKINS_AGENT_NAME} -secret ${JENKINS_SECRET} -workDir ${JENKINS_AGENT_WORKDIR} ${REMOTING_OPTS}

          Probably useless information, but it might just help in triangulation.

          Pay Bas added a comment - - edited We are using our own built agents, which (extremely simplified) boils down to: fedora:40 base image Oracle JDK 17.0.12 remoting jar 3261.v9c670a_4748a_9 In combination with Jenkins LTS 2.462.2 on Fedora 40 + java-11-openjdk-11.0.24.0.8-2.fc40.x86_64 The infrastructure the agent runs on doesn't seem to make a difference. I'm getting the issues both when running our agents through the Jenkins Kubernetes plugin (on OpenShift) as well as simply running the agent as a traditional Docker container on a regular VM. Which leads me to believe this has nothing to do with Kubernetes. Our init script (again extremely simplified): JAVA_OPTS= "-Dhudson.slaves.WorkspaceList= '_' " JNLP_JAVA_OPTIONS= "-Xms512m -Xmx512m -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90" exec ${JAVA17_HOME}/bin/java ${JAVA_OPTS} ${JNLP_JAVA_OPTIONS} -jar /usr/share/jenkins/agent.jar -url ${JENKINS_URL} -name ${JENKINS_AGENT_NAME} -secret ${JENKINS_SECRET} -workDir ${JENKINS_AGENT_WORKDIR} ${REMOTING_OPTS} Probably useless information, but it might just help in triangulation.

          Mark Waite added a comment -

          paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new?

          Mark Waite added a comment - paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new?

          Mark Waite added a comment -

          I've submitted two draft pull requests and would appreciate if those who can duplicate the problem would try them.

          1. Revert the reordering of extension additions https://github.com/jenkinsci/git-plugin/pull/1658
          2. Revert the expansion of a few classes to separate Java objects https://github.com/jenkinsci/git-plugin/pull/1657

          Incremental builds will be available for each of those.

          Mark Waite added a comment - I've submitted two draft pull requests and would appreciate if those who can duplicate the problem would try them. 1. Revert the reordering of extension additions https://github.com/jenkinsci/git-plugin/pull/1658 2. Revert the expansion of a few classes to separate Java objects https://github.com/jenkinsci/git-plugin/pull/1657 Incremental builds will be available for each of those.

          Valentin Delaye added a comment - - edited

          Hi,

          Just step in because I faced the same and needed to urgently rollback ;(

          I can provide more detail on my setup but the error is the same

          returned status code 128:
          [05:43:28.670+02:00] - stdout: 
          [05:43:28.670+02:00] - stderr: Host key verification failed.
          [05:43:28.670+02:00] - fatal: Could not read from remote repository.
          [05:43:28.670+02:00] - 
          [05:43:28.670+02:00] - Please make sure you have the correct access rights
          [05:43:28.670+02:00] - and the repository exists.
          [05:43:28.670+02:00] - 
          [05:43:28.670+02:00] - 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
          [05:43:28.670+02:00] - 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
          [05:43:28.670+02:00] - 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
          [05:43:28.670+02:00] - 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
          

          Rollbacking all plugin with exact same config works

          Valentin Delaye added a comment - - edited Hi, Just step in because I faced the same and needed to urgently rollback ;( I can provide more detail on my setup but the error is the same returned status code 128: [05:43:28.670+02:00] - stdout: [05:43:28.670+02:00] - stderr: Host key verification failed. [05:43:28.670+02:00] - fatal: Could not read from remote repository. [05:43:28.670+02:00] - [05:43:28.670+02:00] - Please make sure you have the correct access rights [05:43:28.670+02:00] - and the repository exists. [05:43:28.670+02:00] - [05:43:28.670+02:00] - at PluginClassLoader for git-client //org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846) [05:43:28.670+02:00] - at PluginClassLoader for git-client //org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185) [05:43:28.670+02:00] - at PluginClassLoader for git-client //org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635) [05:43:28.670+02:00] - at PluginClassLoader for git-client //org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871) Rollbacking all plugin with exact same config works

          List of plugins

          active-directory:2.36
          analysis-model-api:12.5.0
          ansible:403.v8d0ca_dcb_b_502
          ansicolor:1.0.4
          ant:511.v0a_a_1a_334f41b_
          antisamy-markup-formatter:162.v0e6ec0fcfcf6
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f
          artifactory-artifact-manager:122.v0b_d7cb_89ca_34
          artifactory-client-api:2.18.0-35.vee37a_72ccd74
          asm-api:9.7-33.v4d23ef79fcc8
          atlassian-bitbucket-server-integration:4.0.0
          authentication-tokens:1.119.v50285141b_7e1
          badge:2.1
          basic-branch-build-strategies:81.v05e333931c7d
          bitbucket-kubernetes-credentials:336.vc0a_911cde608
          bootstrap5-api:5.3.3-1
          bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
          branch-api:2.1178.v969d9eb_c728e
          build-blocker-plugin:166.vc82fc20b_a_ed6
          build-discarder:139.v05696a_7fe240
          build-name-setter:2.4.3
          build-user-vars-plugin:176.vb_9c7907fd524
          build-with-parameters:76.v9382db_f78962
          byte-buddy-api:1.15.1-59.v81e90f37c4c6
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          checks-api:2.2.1
          cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
          cloudbees-folder:6.951.v5f91d88d76b_b_
          commons-compress-api:1.26.1-2
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-math3-api:3.6.1-4.vdd4613817d74
          commons-text-api:1.12.0-129.v99a_50df237f7
          config-file-provider:978.v8e85886ffdc4
          configuration-as-code:1850.va_a_8c31d3158b_
          coverage-badges-extension:47.v41e62ecf0928
          coverage:1.16.1
          credentials-binding:681.vf91669a_32e45
          credentials:1378.v81ef4269d764
          custom-folder-icon:2.14
          customizable-header:135.vf8ce4237feb_c
          customize-build-now:17.ve5db_875e5343
          dark-theme:479.v661b_1b_911c01
          data-tables-api:2.1.6-1
          database-postgresql:100.v2418e0a_c6909
          database:247.v244b_d85f086d
          display-url-api:2.204.vf6fddd8a_8b_e9
          docker-commons:443.v921729d5611d
          docker-workflow:580.vc0c340686b_54
          dotnet-sdk:1.4.0
          dtkit-api:3.0.2
          durable-task:577.v2a_8a_4b_7c0247
          echarts-api:5.5.1-1
          eddsa-api:0.3.0-4.v84c6f0f4969e
          email-ext:1814.v404722f34263
          embeddable-build-status:487.va_0ef04c898a_2
          extended-read-permission:53.v6499940139e5
          extension-filter:135.v703ff01ca_817
          extra-tool-installers:139.v723fee51b_7f2
          file-operations:266.v9d4e1eb_235b_a_
          file-parameters:339.v4b_cc83e11455
          flatpickr-api:4.6.13-5.v534d8025a_a_59
          flyway-api:9.22.3-151.v475c057b_07fc
          flyway-runner:190.vd433a_679fa_b_8
          font-awesome-api:6.6.0-2
          forensics-api:2.6.0
          generic-tool:1.1
          git-client:5.0.0
          git-forensics:2.2.1
          git-parameter:0.9.19
          git:5.5.0
          github-api:1.321-468.v6a_9f5f2d5a_7e
          github-branch-source:1797.v86fdb_4d57d43
          github-checks:589.v845136f916cd
          github:1.40.0
          gitlab-api:5.6.0-97.v6603a_83f8690
          gitlab-branch-source:710.v6f19df32544b_
          gitlab-kubernetes-credentials:259.v7c72898df530
          gradle:2.13
          gson-api:2.11.0-41.v019fcf6125dc
          h2-api:11.1.4.199-30.v1c64e772f3a_c
          handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
          hidden-parameter:237.v4b_df26c7a_f0e
          htmlpublisher:1.36
          http_request:1.19
          implied-labels:342.vf0a_690315013
          inline-pipeline:1.0.3
          instance-identity:185.v303dc7c645f9
          ionicons-api:74.v93d5eb_813d5f
          jackson2-api:2.17.0-379.v02de8ec9f64c
          jacoco:3.3.6
          jakarta-activation-api:2.1.3-1
          jakarta-mail-api:2.1.3-1
          javadoc:280.v050b_5c849f69
          javax-activation-api:1.2.0-7
          javax-mail-api:1.6.2-10
          jaxb:2.3.9-1
          jdk-tool:80.v8a_dee33ed6f0
          jersey2-api:2.44-151.v6df377fff741
          jjwt-api:0.11.5-112.ve82dfb_224b_a_d
          job-dsl:1.89
          jobcacher-artifactory-storage:51.v06da_175e2655
          jobcacher:551.ve0b_00cb_1b_85c
          joda-time-api:2.13.0-85.vb_64d1c2921f1
          jquery3-api:3.7.1-2
          jsch:0.2.16-86.v42e010d9484b_
          json-api:20240303-41.v94e11e6de726
          json-path-api:2.9.0-58.v62e3e85b_a_655
          junit-attachments:239.v9e003a_c80a_8c
          junit-sql-storage:345.v5094de0a_b_f4d
          junit:1300.v03d9d8a_cf1fb_
          kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
          kubernetes-credentials-provider:1.262.v2670ef7ea_0c5
          kubernetes-credentials:190.v03c305394deb_
          kubernetes:4288.v1719f9d0c854
          ldap:725.v3cb_b_711b_1a_ef
          locale:519.v4e20f313cfa_f
          lockable-resources:1310.v99ca_947ed698
          login-theme:146.v64a_da_cf70ea_6
          mailer:472.vf7c289a_4b_420
          mapdb-api:1.0.9-40.v58107308b_7a_7
          markdown-formatter:220.v1a_262cd9f77f
          mask-passwords:173.v6a_077a_291eb_5
          matrix-auth:3.2.2
          matrix-project:832.va_66e270d2946
          metrics:4.2.21-451.vd51df8df52ec
          mina-sshd-api-common:2.13.2-125.v200281b_61d59
          mina-sshd-api-core:2.13.2-125.v200281b_61d59
          mstest:1.0.5
          next-build-number:1.8
          next-executions:327.v136ff959e97b_
          nodejs:1.6.2
          nunit:485.ve8a_85357320d
          oidc-provider:79.v46f0066a_d813
          okhttp-api:4.11.0-172.vda_da_1feeb_c6e
          openshift-client:1.1.0.424.v829cb_ccf8798
          openshift-k8s-credentials:168.v79a_983191991
          opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4
          oss-symbols-api:71.v08d42ee3785d
          pam-auth:1.11
          parameter-separator:166.vd0120849b_386
          parameterized-scheduler:277.v61a_4b_a_49a_c5c
          parameterized-trigger:806.vf6fff3e28c3e
          pipeline-build-step:540.vb_e8849e1a_b_d8
          pipeline-graph-analysis:216.vfd8b_ece330ca_
          pipeline-graph-view:340.v28cecee8b_25f
          pipeline-groovy-lib:730.ve57b_34648c63
          pipeline-input-step:495.ve9c153f6067b_
          pipeline-maven-api:1421.v610fa_b_e2d60e
          pipeline-maven-database:1421.v610fa_b_e2d60e
          pipeline-maven:1421.v610fa_b_e2d60e
          pipeline-milestone-step:119.vdfdc43fc3b_9a_
          pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
          pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
          pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
          pipeline-npm:204.v4dc4c2202625
          pipeline-rest-api:2.34
          pipeline-stage-step:312.v8cd10304c27a_
          pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
          pipeline-utility-steps:2.17.0
          plain-credentials:183.va_de8f1dd5a_2b_
          platformlabeler:2617.v5444054f5e35
          plugin-util-api:4.1.0
          postgresql-api:42.7.2-40.v76d376d65c77
          postgresql-fingerprint-storage:274.v417e436b_1c5e
          prism-api:1.29.0-17
          prometheus:784.vea_eca_f6592eb_
          pubsub-light:1.18
          purge-build-queue-plugin:88.v23b_97b_f2c7a_d
          resource-disposer:0.23
          scm-api:696.v778d637b_a_762
          scm-filter-branch-pr:148.v0b_5f06e8b_c84
          script-security:1362.v67dc1f0e1b_b_3
          sidebar-link:2.4.1
          simple-theme-plugin:196.v96d9592f4efa_
          skip-notifications-trait:313.vd1337c8f8134
          snakeyaml-api:2.3-123.v13484c65210a_
          sonar:2.17.2
          ssh-agent:376.v8933585c69d3
          ssh-credentials:343.v884f71d78167
          ssh-slaves:2.973.v0fa_8c0dea_f9f
          sshd:3.330.vc866a_8389b_58
          strict-crumb-issuer:2.1.1
          structs:338.v848422169819
          theme-manager:262.vc57ee4a_eda_5d
          timestamper:1.27
          token-macro:400.v35420b_922dcb_
          trilead-api:2.147.vb_73cc728a_32e
          uno-choice:2.8.3
          variant:60.v7290fc0eb_b_cd
          versioncolumn:243.vda_c20eea_a_8a_f
          warnings-ng:11.6.0
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1336.vee415d95c521
          workflow-basic-steps:1058.vcb_fc1e3a_21a_9
          workflow-cps:3964.v0767b_4b_a_0b_fa_
          workflow-durable-task-step:1371.vb_7cec8f3b_95e
          workflow-job:1436.vfa_244484591f
          workflow-multibranch:795.ve0cb_1f45ca_9a_
          workflow-scm-step:427.v4ca_6512e7df1
          workflow-step-api:678.v3ee58b_469476
          workflow-support:926.v9f4f9b_b_98c19
          ws-cleanup:0.46

          List of plugin that doesn't cause issue (1 week difference more less)

          active-directory:2.36
          analysis-model-api:12.5.0
          ansible:403.v8d0ca_dcb_b_502
          ansicolor:1.0.4
          ant:511.v0a_a_1a_334f41b_
          antisamy-markup-formatter:162.v0e6ec0fcfcf6
          apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
          apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f
          artifactory-artifact-manager:122.v0b_d7cb_89ca_34
          artifactory-client-api:2.18.0-35.vee37a_72ccd74
          asm-api:9.7-33.v4d23ef79fcc8
          atlassian-bitbucket-server-integration:4.0.0
          authentication-tokens:1.119.v50285141b_7e1
          badge:2.0
          basic-branch-build-strategies:81.v05e333931c7d
          bitbucket-kubernetes-credentials:336.vc0a_911cde608
          bootstrap5-api:5.3.3-1
          bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
          branch-api:2.1178.v969d9eb_c728e
          build-blocker-plugin:166.vc82fc20b_a_ed6
          build-discarder:139.v05696a_7fe240
          build-name-setter:2.4.3
          build-user-vars-plugin:176.vb_9c7907fd524
          build-with-parameters:76.v9382db_f78962
          byte-buddy-api:1.15.1-59.v81e90f37c4c6
          caffeine-api:3.1.8-133.v17b_1ff2e0599
          checks-api:2.2.1
          cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
          cloudbees-folder:6.951.v5f91d88d76b_b_
          commons-compress-api:1.26.1-2
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-math3-api:3.6.1-4.vdd4613817d74
          commons-text-api:1.12.0-129.v99a_50df237f7
          config-file-provider:978.v8e85886ffdc4
          configuration-as-code:1850.va_a_8c31d3158b_
          coverage-badges-extension:47.v41e62ecf0928
          coverage:1.16.1
          credentials-binding:681.vf91669a_32e45
          credentials:1371.vfee6b_095f0a_3
          custom-folder-icon:2.14
          customizable-header:135.vf8ce4237feb_c
          customize-build-now:17.ve5db_875e5343
          dark-theme:479.v661b_1b_911c01
          data-tables-api:2.1.6-1
          database-postgresql:100.v2418e0a_c6909
          database:247.v244b_d85f086d
          display-url-api:2.204.vf6fddd8a_8b_e9
          docker-commons:443.v921729d5611d
          docker-workflow:580.vc0c340686b_54
          dotnet-sdk:1.4.0
          dtkit-api:3.0.2
          durable-task:577.v2a_8a_4b_7c0247
          echarts-api:5.5.1-1
          eddsa-api:0.3.0-4.v84c6f0f4969e
          email-ext:1814.v404722f34263
          embeddable-build-status:487.va_0ef04c898a_2
          extended-read-permission:53.v6499940139e5
          extension-filter:135.v703ff01ca_817
          extra-tool-installers:139.v723fee51b_7f2
          file-operations:266.v9d4e1eb_235b_a_
          file-parameters:339.v4b_cc83e11455
          flatpickr-api:4.6.13-5.v534d8025a_a_59
          flyway-api:9.22.3-151.v475c057b_07fc
          flyway-runner:190.vd433a_679fa_b_8
          font-awesome-api:6.6.0-2
          forensics-api:2.5.0
          generic-tool:1.1
          git-client:5.0.0
          git-forensics:2.2.1
          git-parameter:0.9.19
          git:5.4.1
          github-api:1.321-468.v6a_9f5f2d5a_7e
          github-branch-source:1797.v86fdb_4d57d43
          github-checks:589.v845136f916cd
          github:1.40.0
          gitlab-api:5.6.0-97.v6603a_83f8690
          gitlab-branch-source:710.v6f19df32544b_
          gitlab-kubernetes-credentials:259.v7c72898df530
          gradle:2.12.1
          gson-api:2.11.0-41.v019fcf6125dc
          h2-api:11.1.4.199-30.v1c64e772f3a_c
          handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
          hidden-parameter:237.v4b_df26c7a_f0e
          htmlpublisher:1.36
          http_request:1.19
          implied-labels:342.vf0a_690315013
          inline-pipeline:1.0.3
          instance-identity:185.v303dc7c645f9
          ionicons-api:74.v93d5eb_813d5f
          jackson2-api:2.17.0-379.v02de8ec9f64c
          jacoco:3.3.6
          jakarta-activation-api:2.1.3-1
          jakarta-mail-api:2.1.3-1
          javadoc:280.v050b_5c849f69
          javax-activation-api:1.2.0-7
          javax-mail-api:1.6.2-10
          jaxb:2.3.9-1
          jdk-tool:80.v8a_dee33ed6f0
          jersey2-api:2.44-151.v6df377fff741
          jjwt-api:0.11.5-112.ve82dfb_224b_a_d
          job-dsl:1.89
          jobcacher-artifactory-storage:51.v06da_175e2655
          jobcacher:551.ve0b_00cb_1b_85c
          joda-time-api:2.13.0-85.vb_64d1c2921f1
          jquery3-api:3.7.1-2
          jsch:0.2.16-86.v42e010d9484b_
          json-api:20240303-41.v94e11e6de726
          json-path-api:2.9.0-58.v62e3e85b_a_655
          junit-attachments:239.v9e003a_c80a_8c
          junit-sql-storage:345.v5094de0a_b_f4d
          junit:1300.v03d9d8a_cf1fb_
          kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
          kubernetes-credentials-provider:1.262.v2670ef7ea_0c5
          kubernetes-credentials:189.v90a_488b_d1d65
          kubernetes:4287.v73451380b_576
          ldap:725.v3cb_b_711b_1a_ef
          locale:519.v4e20f313cfa_f
          lockable-resources:1301.v0e3b_da_4b_4462
          login-theme:146.v64a_da_cf70ea_6
          mailer:472.vf7c289a_4b_420
          mapdb-api:1.0.9-40.v58107308b_7a_7
          markdown-formatter:220.v1a_262cd9f77f
          mask-passwords:173.v6a_077a_291eb_5
          matrix-auth:3.2.2
          matrix-project:832.va_66e270d2946
          metrics:4.2.21-451.vd51df8df52ec
          mina-sshd-api-common:2.13.2-125.v200281b_61d59
          mina-sshd-api-core:2.13.2-125.v200281b_61d59
          mstest:1.0.5
          next-build-number:1.8
          next-executions:327.v136ff959e97b_
          nodejs:1.6.2
          nunit:485.ve8a_85357320d
          oidc-provider:79.v46f0066a_d813
          okhttp-api:4.11.0-172.vda_da_1feeb_c6e
          openshift-client:1.1.0.424.v829cb_ccf8798
          openshift-k8s-credentials:168.v79a_983191991
          opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4
          oss-symbols-api:67.v6cc456b_ed2fb_
          pam-auth:1.11
          parameter-separator:166.vd0120849b_386
          parameterized-scheduler:277.v61a_4b_a_49a_c5c
          parameterized-trigger:806.vf6fff3e28c3e
          pipeline-build-step:540.vb_e8849e1a_b_d8
          pipeline-graph-analysis:216.vfd8b_ece330ca_
          pipeline-graph-view:340.v28cecee8b_25f
          pipeline-groovy-lib:730.ve57b_34648c63
          pipeline-input-step:495.ve9c153f6067b_
          pipeline-maven-api:1421.v610fa_b_e2d60e
          pipeline-maven-database:1421.v610fa_b_e2d60e
          pipeline-maven:1421.v610fa_b_e2d60e
          pipeline-milestone-step:119.vdfdc43fc3b_9a_
          pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
          pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
          pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
          pipeline-npm:204.v4dc4c2202625
          pipeline-rest-api:2.34
          pipeline-stage-step:312.v8cd10304c27a_
          pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
          pipeline-utility-steps:2.17.0
          plain-credentials:183.va_de8f1dd5a_2b_
          platformlabeler:2617.v5444054f5e35
          plugin-util-api:4.1.0
          postgresql-api:42.7.2-40.v76d376d65c77
          postgresql-fingerprint-storage:274.v417e436b_1c5e
          prism-api:1.29.0-17
          prometheus:784.vea_eca_f6592eb_
          pubsub-light:1.18
          purge-build-queue-plugin:88.v23b_97b_f2c7a_d
          resource-disposer:0.23
          scm-api:696.v778d637b_a_762
          scm-filter-branch-pr:148.v0b_5f06e8b_c84
          script-security:1362.v67dc1f0e1b_b_3
          sidebar-link:2.4.1
          simple-theme-plugin:196.v96d9592f4efa_
          skip-notifications-trait:313.vd1337c8f8134
          snakeyaml-api:2.3-123.v13484c65210a_
          sonar:2.17.2
          ssh-agent:376.v8933585c69d3
          ssh-credentials:343.v884f71d78167
          ssh-slaves:2.973.v0fa_8c0dea_f9f
          sshd:3.330.vc866a_8389b_58
          strict-crumb-issuer:2.1.1
          structs:338.v848422169819
          theme-manager:262.vc57ee4a_eda_5d
          timestamper:1.27
          token-macro:400.v35420b_922dcb_
          trilead-api:2.147.vb_73cc728a_32e
          uno-choice:2.8.3
          variant:60.v7290fc0eb_b_cd
          versioncolumn:243.vda_c20eea_a_8a_f
          warnings-ng:11.6.0
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1336.vee415d95c521
          workflow-basic-steps:1058.vcb_fc1e3a_21a_9
          workflow-cps:3961.ve48ee2c44a_b_3
          workflow-durable-task-step:1371.vb_7cec8f3b_95e
          workflow-job:1436.vfa_244484591f
          workflow-multibranch:795.ve0cb_1f45ca_9a_
          workflow-scm-step:427.v4ca_6512e7df1
          workflow-step-api:678.v3ee58b_469476
          workflow-support:920.v59f71ce16f04
          ws-cleanup:0.46
          

          The error doesn't seem to arrive in all condition

          • Checking out global libraries on the controller seems to work
          • Explicit git checkout in a scripted pipeline seems to work
          • Implicit git checkout in a declararive pipeline doesn't work in a multibranch

          Hope it can help

          Valentin Delaye added a comment - List of plugins active-directory:2.36 analysis-model-api:12.5.0 ansible:403.v8d0ca_dcb_b_502 ansicolor:1.0.4 ant:511.v0a_a_1a_334f41b_ antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f artifactory-artifact-manager:122.v0b_d7cb_89ca_34 artifactory-client-api:2.18.0-35.vee37a_72ccd74 asm-api:9.7-33.v4d23ef79fcc8 atlassian-bitbucket-server-integration:4.0.0 authentication-tokens:1.119.v50285141b_7e1 badge:2.1 basic-branch-build-strategies:81.v05e333931c7d bitbucket-kubernetes-credentials:336.vc0a_911cde608 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_ branch-api:2.1178.v969d9eb_c728e build-blocker-plugin:166.vc82fc20b_a_ed6 build-discarder:139.v05696a_7fe240 build-name-setter:2.4.3 build-user-vars-plugin:176.vb_9c7907fd524 build-with-parameters:76.v9382db_f78962 byte-buddy-api:1.15.1-59.v81e90f37c4c6 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.1 cloudbees-bitbucket-branch-source:888.v8e6d479a_1730 cloudbees-folder:6.951.v5f91d88d76b_b_ commons-compress-api:1.26.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-math3-api:3.6.1-4.vdd4613817d74 commons-text-api:1.12.0-129.v99a_50df237f7 config-file-provider:978.v8e85886ffdc4 configuration-as-code:1850.va_a_8c31d3158b_ coverage-badges-extension:47.v41e62ecf0928 coverage:1.16.1 credentials-binding:681.vf91669a_32e45 credentials:1378.v81ef4269d764 custom-folder-icon:2.14 customizable-header:135.vf8ce4237feb_c customize-build-now:17.ve5db_875e5343 dark-theme:479.v661b_1b_911c01 data-tables-api:2.1.6-1 database-postgresql:100.v2418e0a_c6909 database:247.v244b_d85f086d display-url-api:2.204.vf6fddd8a_8b_e9 docker-commons:443.v921729d5611d docker-workflow:580.vc0c340686b_54 dotnet-sdk:1.4.0 dtkit-api:3.0.2 durable-task:577.v2a_8a_4b_7c0247 echarts-api:5.5.1-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1814.v404722f34263 embeddable-build-status:487.va_0ef04c898a_2 extended-read-permission:53.v6499940139e5 extension-filter:135.v703ff01ca_817 extra-tool-installers:139.v723fee51b_7f2 file-operations:266.v9d4e1eb_235b_a_ file-parameters:339.v4b_cc83e11455 flatpickr-api:4.6.13-5.v534d8025a_a_59 flyway-api:9.22.3-151.v475c057b_07fc flyway-runner:190.vd433a_679fa_b_8 font-awesome-api:6.6.0-2 forensics-api:2.6.0 generic-tool:1.1 git-client:5.0.0 git-forensics:2.2.1 git-parameter:0.9.19 git:5.5.0 github-api:1.321-468.v6a_9f5f2d5a_7e github-branch-source:1797.v86fdb_4d57d43 github-checks:589.v845136f916cd github:1.40.0 gitlab-api:5.6.0-97.v6603a_83f8690 gitlab-branch-source:710.v6f19df32544b_ gitlab-kubernetes-credentials:259.v7c72898df530 gradle:2.13 gson-api:2.11.0-41.v019fcf6125dc h2-api:11.1.4.199-30.v1c64e772f3a_c handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:237.v4b_df26c7a_f0e htmlpublisher:1.36 http_request:1.19 implied-labels:342.vf0a_690315013 inline-pipeline:1.0.3 instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jacoco:3.3.6 jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:280.v050b_5c849f69 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:80.v8a_dee33ed6f0 jersey2-api:2.44-151.v6df377fff741 jjwt-api:0.11.5-112.ve82dfb_224b_a_d job-dsl:1.89 jobcacher-artifactory-storage:51.v06da_175e2655 jobcacher:551.ve0b_00cb_1b_85c joda-time-api:2.13.0-85.vb_64d1c2921f1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit-attachments:239.v9e003a_c80a_8c junit-sql-storage:345.v5094de0a_b_f4d junit:1300.v03d9d8a_cf1fb_ kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials-provider:1.262.v2670ef7ea_0c5 kubernetes-credentials:190.v03c305394deb_ kubernetes:4288.v1719f9d0c854 ldap:725.v3cb_b_711b_1a_ef locale:519.v4e20f313cfa_f lockable-resources:1310.v99ca_947ed698 login-theme:146.v64a_da_cf70ea_6 mailer:472.vf7c289a_4b_420 mapdb-api:1.0.9-40.v58107308b_7a_7 markdown-formatter:220.v1a_262cd9f77f mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.2 matrix-project:832.va_66e270d2946 metrics:4.2.21-451.vd51df8df52ec mina-sshd-api-common:2.13.2-125.v200281b_61d59 mina-sshd-api-core:2.13.2-125.v200281b_61d59 mstest:1.0.5 next-build-number:1.8 next-executions:327.v136ff959e97b_ nodejs:1.6.2 nunit:485.ve8a_85357320d oidc-provider:79.v46f0066a_d813 okhttp-api:4.11.0-172.vda_da_1feeb_c6e openshift-client:1.1.0.424.v829cb_ccf8798 openshift-k8s-credentials:168.v79a_983191991 opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4 oss-symbols-api:71.v08d42ee3785d pam-auth:1.11 parameter-separator:166.vd0120849b_386 parameterized-scheduler:277.v61a_4b_a_49a_c5c parameterized-trigger:806.vf6fff3e28c3e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-graph-view:340.v28cecee8b_25f pipeline-groovy-lib:730.ve57b_34648c63 pipeline-input-step:495.ve9c153f6067b_ pipeline-maven-api:1421.v610fa_b_e2d60e pipeline-maven-database:1421.v610fa_b_e2d60e pipeline-maven:1421.v610fa_b_e2d60e pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83 pipeline-npm:204.v4dc4c2202625 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83 pipeline-utility-steps:2.17.0 plain-credentials:183.va_de8f1dd5a_2b_ platformlabeler:2617.v5444054f5e35 plugin-util-api:4.1.0 postgresql-api:42.7.2-40.v76d376d65c77 postgresql-fingerprint-storage:274.v417e436b_1c5e prism-api:1.29.0-17 prometheus:784.vea_eca_f6592eb_ pubsub-light:1.18 purge-build-queue-plugin:88.v23b_97b_f2c7a_d resource-disposer:0.23 scm-api:696.v778d637b_a_762 scm-filter-branch-pr:148.v0b_5f06e8b_c84 script-security:1362.v67dc1f0e1b_b_3 sidebar-link:2.4.1 simple-theme-plugin:196.v96d9592f4efa_ skip-notifications-trait:313.vd1337c8f8134 snakeyaml-api:2.3-123.v13484c65210a_ sonar:2.17.2 ssh-agent:376.v8933585c69d3 ssh-credentials:343.v884f71d78167 ssh-slaves:2.973.v0fa_8c0dea_f9f sshd:3.330.vc866a_8389b_58 strict-crumb-issuer:2.1.1 structs:338.v848422169819 theme-manager:262.vc57ee4a_eda_5d timestamper:1.27 token-macro:400.v35420b_922dcb_ trilead-api:2.147.vb_73cc728a_32e uno-choice:2.8.3 variant:60.v7290fc0eb_b_cd versioncolumn:243.vda_c20eea_a_8a_f warnings-ng:11.6.0 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1336.vee415d95c521 workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3964.v0767b_4b_a_0b_fa_ workflow-durable-task-step:1371.vb_7cec8f3b_95e workflow-job:1436.vfa_244484591f workflow-multibranch:795.ve0cb_1f45ca_9a_ workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:678.v3ee58b_469476 workflow-support:926.v9f4f9b_b_98c19 ws-cleanup:0.46 List of plugin that doesn't cause issue (1 week difference more less) active-directory:2.36 analysis-model-api:12.5.0 ansible:403.v8d0ca_dcb_b_502 ansicolor:1.0.4 ant:511.v0a_a_1a_334f41b_ antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f artifactory-artifact-manager:122.v0b_d7cb_89ca_34 artifactory-client-api:2.18.0-35.vee37a_72ccd74 asm-api:9.7-33.v4d23ef79fcc8 atlassian-bitbucket-server-integration:4.0.0 authentication-tokens:1.119.v50285141b_7e1 badge:2.0 basic-branch-build-strategies:81.v05e333931c7d bitbucket-kubernetes-credentials:336.vc0a_911cde608 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_ branch-api:2.1178.v969d9eb_c728e build-blocker-plugin:166.vc82fc20b_a_ed6 build-discarder:139.v05696a_7fe240 build-name-setter:2.4.3 build-user-vars-plugin:176.vb_9c7907fd524 build-with-parameters:76.v9382db_f78962 byte -buddy-api:1.15.1-59.v81e90f37c4c6 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.1 cloudbees-bitbucket-branch-source:888.v8e6d479a_1730 cloudbees-folder:6.951.v5f91d88d76b_b_ commons-compress-api:1.26.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-math3-api:3.6.1-4.vdd4613817d74 commons-text-api:1.12.0-129.v99a_50df237f7 config-file-provider:978.v8e85886ffdc4 configuration-as-code:1850.va_a_8c31d3158b_ coverage-badges-extension:47.v41e62ecf0928 coverage:1.16.1 credentials-binding:681.vf91669a_32e45 credentials:1371.vfee6b_095f0a_3 custom-folder-icon:2.14 customizable-header:135.vf8ce4237feb_c customize-build-now:17.ve5db_875e5343 dark-theme:479.v661b_1b_911c01 data-tables-api:2.1.6-1 database-postgresql:100.v2418e0a_c6909 database:247.v244b_d85f086d display-url-api:2.204.vf6fddd8a_8b_e9 docker-commons:443.v921729d5611d docker-workflow:580.vc0c340686b_54 dotnet-sdk:1.4.0 dtkit-api:3.0.2 durable-task:577.v2a_8a_4b_7c0247 echarts-api:5.5.1-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1814.v404722f34263 embeddable-build-status:487.va_0ef04c898a_2 extended-read-permission:53.v6499940139e5 extension-filter:135.v703ff01ca_817 extra-tool-installers:139.v723fee51b_7f2 file-operations:266.v9d4e1eb_235b_a_ file-parameters:339.v4b_cc83e11455 flatpickr-api:4.6.13-5.v534d8025a_a_59 flyway-api:9.22.3-151.v475c057b_07fc flyway-runner:190.vd433a_679fa_b_8 font-awesome-api:6.6.0-2 forensics-api:2.5.0 generic -tool:1.1 git-client:5.0.0 git-forensics:2.2.1 git-parameter:0.9.19 git:5.4.1 github-api:1.321-468.v6a_9f5f2d5a_7e github-branch-source:1797.v86fdb_4d57d43 github-checks:589.v845136f916cd github:1.40.0 gitlab-api:5.6.0-97.v6603a_83f8690 gitlab-branch-source:710.v6f19df32544b_ gitlab-kubernetes-credentials:259.v7c72898df530 gradle:2.12.1 gson-api:2.11.0-41.v019fcf6125dc h2-api:11.1.4.199-30.v1c64e772f3a_c handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:237.v4b_df26c7a_f0e htmlpublisher:1.36 http_request:1.19 implied-labels:342.vf0a_690315013 inline-pipeline:1.0.3 instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jacoco:3.3.6 jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:280.v050b_5c849f69 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:80.v8a_dee33ed6f0 jersey2-api:2.44-151.v6df377fff741 jjwt-api:0.11.5-112.ve82dfb_224b_a_d job-dsl:1.89 jobcacher-artifactory-storage:51.v06da_175e2655 jobcacher:551.ve0b_00cb_1b_85c joda-time-api:2.13.0-85.vb_64d1c2921f1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit-attachments:239.v9e003a_c80a_8c junit-sql-storage:345.v5094de0a_b_f4d junit:1300.v03d9d8a_cf1fb_ kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials-provider:1.262.v2670ef7ea_0c5 kubernetes-credentials:189.v90a_488b_d1d65 kubernetes:4287.v73451380b_576 ldap:725.v3cb_b_711b_1a_ef locale:519.v4e20f313cfa_f lockable-resources:1301.v0e3b_da_4b_4462 login-theme:146.v64a_da_cf70ea_6 mailer:472.vf7c289a_4b_420 mapdb-api:1.0.9-40.v58107308b_7a_7 markdown-formatter:220.v1a_262cd9f77f mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.2 matrix-project:832.va_66e270d2946 metrics:4.2.21-451.vd51df8df52ec mina-sshd-api-common:2.13.2-125.v200281b_61d59 mina-sshd-api-core:2.13.2-125.v200281b_61d59 mstest:1.0.5 next-build-number:1.8 next-executions:327.v136ff959e97b_ nodejs:1.6.2 nunit:485.ve8a_85357320d oidc-provider:79.v46f0066a_d813 okhttp-api:4.11.0-172.vda_da_1feeb_c6e openshift-client:1.1.0.424.v829cb_ccf8798 openshift-k8s-credentials:168.v79a_983191991 opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4 oss-symbols-api:67.v6cc456b_ed2fb_ pam-auth:1.11 parameter-separator:166.vd0120849b_386 parameterized-scheduler:277.v61a_4b_a_49a_c5c parameterized-trigger:806.vf6fff3e28c3e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-graph-view:340.v28cecee8b_25f pipeline-groovy-lib:730.ve57b_34648c63 pipeline-input-step:495.ve9c153f6067b_ pipeline-maven-api:1421.v610fa_b_e2d60e pipeline-maven-database:1421.v610fa_b_e2d60e pipeline-maven:1421.v610fa_b_e2d60e pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83 pipeline-npm:204.v4dc4c2202625 pipeline- rest -api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83 pipeline-utility-steps:2.17.0 plain-credentials:183.va_de8f1dd5a_2b_ platformlabeler:2617.v5444054f5e35 plugin-util-api:4.1.0 postgresql-api:42.7.2-40.v76d376d65c77 postgresql-fingerprint-storage:274.v417e436b_1c5e prism-api:1.29.0-17 prometheus:784.vea_eca_f6592eb_ pubsub-light:1.18 purge-build-queue-plugin:88.v23b_97b_f2c7a_d resource-disposer:0.23 scm-api:696.v778d637b_a_762 scm-filter-branch-pr:148.v0b_5f06e8b_c84 script-security:1362.v67dc1f0e1b_b_3 sidebar-link:2.4.1 simple-theme-plugin:196.v96d9592f4efa_ skip-notifications-trait:313.vd1337c8f8134 snakeyaml-api:2.3-123.v13484c65210a_ sonar:2.17.2 ssh-agent:376.v8933585c69d3 ssh-credentials:343.v884f71d78167 ssh-slaves:2.973.v0fa_8c0dea_f9f sshd:3.330.vc866a_8389b_58 strict-crumb-issuer:2.1.1 structs:338.v848422169819 theme-manager:262.vc57ee4a_eda_5d timestamper:1.27 token-macro:400.v35420b_922dcb_ trilead-api:2.147.vb_73cc728a_32e uno-choice:2.8.3 variant:60.v7290fc0eb_b_cd versioncolumn:243.vda_c20eea_a_8a_f warnings-ng:11.6.0 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1336.vee415d95c521 workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3961.ve48ee2c44a_b_3 workflow-durable-task-step:1371.vb_7cec8f3b_95e workflow-job:1436.vfa_244484591f workflow-multibranch:795.ve0cb_1f45ca_9a_ workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:678.v3ee58b_469476 workflow-support:920.v59f71ce16f04 ws-cleanup:0.46 The error doesn't seem to arrive in all condition Checking out global libraries on the controller seems to work Explicit git checkout in a scripted pipeline seems to work Implicit git checkout in a declararive pipeline doesn't work in a multibranch Hope it can help

          Yoerg added a comment -

          Tried it:

          • PR-1657 Host key verification failed.
          • PR-1658 Verifying host key using manually-configured host key entries

          Yoerg added a comment - Tried it: PR-1657 Host key verification failed. PR-1658 Verifying host key using manually-configured host key entries

          Yoerg added a comment - - edited

          I believe this might be related specifically to Bitbucket multibranch pipelines? I've tried the exact same pipeline with a GitHub multibranch, a Git multibranch and a Bitbucket multibranch pipeline and only the Bitbucket pipeline fails!

          Affected uses, can you report your source server type and job setup?

          Yoerg added a comment - - edited I believe this might be related specifically to Bitbucket multibranch pipelines ? I've tried the exact same pipeline with a GitHub multibranch, a Git multibranch and a Bitbucket multibranch pipeline and only the Bitbucket pipeline fails! Affected uses, can you report your source server type and job setup?

          Pay Bas added a comment - - edited

          paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new?

          markewaite we are using "Git Host Key Verification Configuration" = Manually provided keys

          yoerg I downgraded pretty quickly but looking back at the failed jobs it does indeed seem to suggest that regular pipelines were unaffected but Bitbucket multibranch pipelines which use https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ were affected.

          I have no non-Bitbucket multibranch pipelines to compare with however.

          Pay Bas added a comment - - edited paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new? markewaite we are using "Git Host Key Verification Configuration" = Manually provided keys yoerg I downgraded pretty quickly but looking back at the failed jobs it does indeed seem to suggest that regular pipelines were unaffected but Bitbucket multibranch pipelines which use https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ were affected. I have no non-Bitbucket multibranch pipelines to compare with however.

          Ruby Paasche added a comment -

          yoerg

          Yes, in fact, is Bitbucket DC with Bitbucket multibranch.

          Ruby Paasche added a comment - yoerg Yes, in fact, is Bitbucket DC with Bitbucket multibranch.

          Same here. Multibranch bitbucket pipeline seems only affected. Not regular ones

          Valentin Delaye added a comment - Same here. Multibranch bitbucket pipeline seems only affected. Not regular ones

          Ruby Paasche added a comment - - edited

          yoerg markewaite

          Looks like the BitBucket multibranch plugin overrides the
          auth and with "reordering of extension additions" the Creds from the Bitbucket multibranch plugin gets active and not the one from the git plugin.

          Ruby Paasche added a comment - - edited yoerg markewaite Looks like the BitBucket multibranch plugin overrides the auth and with "reordering of extension additions" the Creds from the Bitbucket multibranch plugin gets active and not the one from the git plugin.

          Ruby Paasche added a comment -

          Tried it:

          PR-1657 Host key verification failed.
          PR-1658 Verifying host key using manually-configured host key entries

          Can confirm this.

          Ruby Paasche added a comment - Tried it: PR-1657 Host key verification failed. PR-1658 Verifying host key using manually-configured host key entries Can confirm this.

          Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Valentin Delaye added a comment - Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Daniel added a comment - - edited

          I found another workaround beside downgrading the plugin. 

          1.) 
          In case you use the Bitbucket multibranch setup, do not checkout via ssh anymore, use the PAT token instead. 

          With the ssh checkout, we had the effect that all pull request jobs directly fail. Non pull request jobs were somehow not affected. Without the ssh checkout, the pull request jobs start again. 

           

          2.) In case that you invoke the checkout step directly in your Jenkinsfile, consider the following.
               Try to explicitly inject the credentials instead of relying on the scm variable.
               So, instead of:

              gitSCM = checkout(
                [
                  $class                           : 'GitSCM',
                  branches                         : scm.branches,
                  doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
                  userRemoteConfigs                : scm.userRemoteConfigs,
                  extensions                       : [
                    [$class   : "CloneOption",
                     noTags   : noTags,
                     shallow  : isShallowClone,
                     depth    : cloneDepth,
                     reference: ''],
                  ],
                ],
              )

          Use something like:

              gitSCM = checkout(
                [
                  $class                           : 'GitSCM',
                  branches                         : scm.branches,
                  doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
                  userRemoteConfigs                : [[url: scm.userRemoteConfigs[0].url, credentialsId: "YourCredentialsID",
                  extensions                       : [
                    [$class   : "CloneOption",
                     noTags   : noTags,
                     shallow  : isShallowClone,
                     depth    : cloneDepth,
                     reference: ''],
                  ],
                ],
              )

           

          Maybe this is helpful for somebody.

          Daniel added a comment - - edited I found another workaround beside downgrading the plugin.  1.)  In case you use the Bitbucket multibranch setup, do not checkout via ssh anymore, use the PAT token instead.  With the ssh checkout, we had the effect that all pull request jobs directly fail. Non pull request jobs were somehow not affected. Without the ssh checkout, the pull request jobs start again.    2.) In case that you invoke the checkout step directly in your Jenkinsfile, consider the following.      Try to explicitly inject the credentials instead of relying on the scm variable.      So, instead of:     gitSCM = checkout(       [         $class                           : 'GitSCM',         branches                         : scm.branches,         doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,         userRemoteConfigs                : scm.userRemoteConfigs,         extensions                       : [           [$class   : "CloneOption",            noTags   : noTags,            shallow  : isShallowClone,            depth    : cloneDepth,            reference: ''],         ],       ],     ) Use something like:     gitSCM = checkout(       [         $class                           : 'GitSCM',         branches                         : scm.branches,         doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,         userRemoteConfigs                : [[url: scm.userRemoteConfigs [0] .url, credentialsId: "YourCredentialsID",         extensions                       : [           [$class   : "CloneOption",            noTags   : noTags,            shallow  : isShallowClone,            depth    : cloneDepth,            reference: ''],         ],       ],     )   M aybe this is helpful for somebody.

          Mark Waite added a comment -

          Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?

          Thanks for the suggestion. I'm releasing git plugin 5.5.1 that removes the pull request so that I then have time to implement the change with appropriate test environment improvements that will detect this type of failure. I believe that my test environment already has the ability to show this failure, but I didn't check the status of every test job after making that change. I knew the change was risky and had evaluated it in several different ways (my environment, plugin tests, plugin BOM, acceptance test harness), but did not specifically check the behavior of Bitbucket branch source plugin multibranch Pipelines with ssh authentication. I believe that I have those job types, but I did not check their status.

          Mark Waite added a comment - Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ? Thanks for the suggestion. I'm releasing git plugin 5.5.1 that removes the pull request so that I then have time to implement the change with appropriate test environment improvements that will detect this type of failure. I believe that my test environment already has the ability to show this failure, but I didn't check the status of every test job after making that change. I knew the change was risky and had evaluated it in several different ways (my environment, plugin tests, plugin BOM, acceptance test harness), but did not specifically check the behavior of Bitbucket branch source plugin multibranch Pipelines with ssh authentication. I believe that I have those job types, but I did not check their status.

          Mark Waite added a comment -

          I've released git plugin 5.5.1 that reverts the reordering of extension additions. My sincere apologies to everyone affected by the issue.

          I'll work on additional automation that detects that type of failure during pull request evaluation, since the additional automation will help many other pull requests as well.

          Mark Waite added a comment - I've released git plugin 5.5.1 that reverts the reordering of extension additions. My sincere apologies to everyone affected by the issue. I'll work on additional automation that detects that type of failure during pull request evaluation, since the additional automation will help many other pull requests as well.

          Pay Bas added a comment -

          My sincere apologies to everyone affected by the issue.

          I don't see why you should apologize. This is an open-source project maintained largely by volunteers and we're all using it for free.

          Besides: the ecosystem is massive, so every possible plugin interaction is impossible to test beforehand.

          We should be thanking you for your dedication and rapid response.

          Pay Bas added a comment - My sincere apologies to everyone affected by the issue. I don't see why you should apologize. This is an open-source project maintained largely by volunteers and we're all using it for free. Besides: the ecosystem is massive, so every possible plugin interaction is impossible to test beforehand. We should be thanking you for your dedication and rapid response.

          It's not just host key verification, actually testing this the GitSCM was using the Pipeline Multibranch scan credentials instead of the SSH Credentials (that could check in the build log that is not showing using GIT_SSH anymore. Bitbucket Branch source in the scenario of SSHCheckoutTrait configures a GitSCM with the SSH credentials but also an authenticator extension (supposed to decorate the GitClient with inferred authentication from the Pipeline Multibranch scan credentials). Which is probably wrong.
          The way authentication is pre-configures is something I was trying to address as part of https://issues.jenkins.io/browse/JENKINS-73471. I have added a test for this scenario in Bitbucket Branch Source to cover this https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/867/files. Hopefully this covers it for the next attempt to apply JENKINS-73677

          Allan BURDAJEWICZ added a comment - It's not just host key verification, actually testing this the GitSCM was using the Pipeline Multibranch scan credentials instead of the SSH Credentials (that could check in the build log that is not showing using GIT_SSH anymore. Bitbucket Branch source in the scenario of SSHCheckoutTrait configures a GitSCM with the SSH credentials but also an authenticator extension (supposed to decorate the GitClient with inferred authentication from the Pipeline Multibranch scan credentials). Which is probably wrong. The way authentication is pre-configures is something I was trying to address as part of https://issues.jenkins.io/browse/JENKINS-73471 . I have added a test for this scenario in Bitbucket Branch Source to cover this https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/867/files . Hopefully this covers it for the next attempt to apply JENKINS-73677

          Should further be prevented by bitbucket-branch-source-plugin:906.vedf430cb_4481.

          Allan BURDAJEWICZ added a comment - Should further be prevented by bitbucket-branch-source-plugin:906.vedf430cb_4481.

            markewaite Mark Waite
            rpaasche Ruby Paasche
            Votes:
            6 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: