-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Kubernetes
GIT-Plugin 5.5.0
Since release 5.5.0 of the Git-Client the "security" → "Git Host Key Verification Configuration" setting is ignored (at least on Kubernetes Agents).
Workaround is manual downgrade to 5.4.1
Using anything other than "Known hosts files" causes "stderr: Host key verification failed." followed by "fatal: Could not read from remote repository."
Checking out the pipeline repos beforehand is still working, seems to be a problem on kubernetes agents builds only.
- sysinfo.txt
- 156 kB
- run-jenkins.sh
- 0.9 kB
- plugins.txt
- 2 kB
- blocks
-
JENKINS-73677 Initialize GitClient credentials before decorating it with extensions
-
- Closed
-
- is related to
-
JENKINS-73803 Checkout over ssh no longer works after upgrading to 5.5.0
-
- Resolved
-
[JENKINS-73797] Git checkout of Bitbucket multibranch Pipelines fails when using ssh
Thanks for reporting the issue rpaasche.
Can you confirm that when you upgraded to git plugin 5.5.0 that you also upgraded git client plugin? The git client plugin handles the host key verification configuration. There were host key verification changes in git client plugin 5.0.0 If you were running git client plugin 4.7.0 before and upgraded to git client plugin 5.0.0, that would narrow the search range for my investigation.
You mention that it is specific to kubernetes agents. Can you share the other agent types that you have tried?
Can you share the list of installed plugins before the problem and the list of installed plugins after the problem appeared? "How to report an issue" includes a script that generates the list of installed plugins on a running system.
Thanks yoerg. Same questions apply to your comment. Can you provide the detailed list of plugins installed before the problem was seen and after the problem was seen?
Right, sorry: Jenkins LTS 2.462.2 on Java 21.0.4+7-Ubuntu-1ubuntu222.04
I didn't change anything besides the Git plugin.
Working configuration:
- sysinfo.txt
- Git client plugin 5.0.0
- Git plugin 5.4.1
Failing configuration:
- Git client plugin 5.0.0
- Git plugin 5.5.0
Git client plugin 6 requires Jenkins 2.463 so it's not available for the current LTS
Can confirm. Also had to downgrade to 5.4.1 due to:
ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- ssh://git@bitbucket.mega.corp:7999/project/repo.git +refs/heads/task/PROJ-83195-my-branch:refs/remotes/origin/task/PROJ-83195-my-branch +refs/heads/develop:refs/remotes/origin/develop" returned status code 128: stdout: stderr: git@bitbucket.mega.corp: Permission denied (publickey). fatal: Could not read from remote repository.
I am having the same problem ... agent is a AWS cloud agent, not k8s.
- Git client plugin 5.0.0
- Git plugin 5.5.0
- Jenkins 2.462.2 (LTS) on Ubuntu
Git-Client 5.0.0 + Git 5.5.0 ⇾ doesn't work
Git-Client 5.0.0 + Git 5.4.1 ⇾ works
Git-Client 6.0.0 + Git 5.4.1 ⇾ causes `java.lang.NullPointerException: Cannot invoke "jenkins.scm.api.SCMHead.getName()" because "this.head" is null` (but was expected, we are on the LTS version)
Git-Client 6.0.0 + Git 5.5.0 ⇾ doesn't work
Thanks for the additional reports. That further confirms there is a real problem, though I don't yet know the source of the problem.
I'm not able to duplicate the problem in my configuration testing. I'm using a static agent in various forms rather than using a Kubernetes agent. I would really appreciate additional details that would allow me to duplicate it, especially if it can be duplicated without requiring that I install and configure a Kubernetes cluster.
Steps that I've attempted while trying to duplicate the problem:
- Create a plugins.txt
file that defines the plugins to be used
- Create a run-jenkins.sh
script that downloads Jenkins 2.462.2 and the specified plugin versions
- Run the run-jenkins.sh
script and complete the setup wizard by creating a user and choosing to install no additional plugins
- Configure the controller with 0 executors and define an inbound agent
- Create a new user account on a Linux computer and connect the inbound agent with that account (so that it has no values for known hosts)
- Set the host key configuration to use manually configured host keys and add the host key for GitHub.com
- Define a Jenkins private key credential that is known to my user account on GitHub.com
- Define a Jenkins job that clones from a private GitHub.com repository and uses the define private key credential
- Run the job and confirm that it is successful
- Remove the host key for GitHub.com from the manually configured host keys on the Jenkins controller
- Run the job and confirm that it fails because the host key is not available
Additional testing with an Alpine agent from the docker plugin using an Alpine does not show the issue either. I confirmed that manually provided host keys are sent to the Alpine agent that is started by the docker plugin. The Alpine agent that I'm using is the most recent release https://hub.docker.com/r/jenkins/inbound-agent/tags?page_size=&ordering=&name=3261.v9c670a_4748a_9-4-alpine
We are using our own built agents, which (extremely simplified) boils down to:
- fedora:40 base image
- Oracle JDK 17.0.12
- remoting jar 3261.v9c670a_4748a_9
In combination with Jenkins LTS 2.462.2 on Fedora 40 + java-11-openjdk-11.0.24.0.8-2.fc40.x86_64
The infrastructure the agent runs on doesn't seem to make a difference. I'm getting the issues both when running our agents through the Jenkins Kubernetes plugin (on OpenShift) as well as simply running the agent as a traditional Docker container on a regular VM. Which leads me to believe this has nothing to do with Kubernetes.
Our init script (again extremely simplified):
JAVA_OPTS="-Dhudson.slaves.WorkspaceList='_'" JNLP_JAVA_OPTIONS="-Xms512m -Xmx512m -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90" exec ${JAVA17_HOME}/bin/java ${JAVA_OPTS} ${JNLP_JAVA_OPTIONS} -jar /usr/share/jenkins/agent.jar -url ${JENKINS_URL} -name ${JENKINS_AGENT_NAME} -secret ${JENKINS_SECRET} -workDir ${JENKINS_AGENT_WORKDIR} ${REMOTING_OPTS}
Probably useless information, but it might just help in triangulation.
paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new?
I've submitted two draft pull requests and would appreciate if those who can duplicate the problem would try them.
1. Revert the reordering of extension additions https://github.com/jenkinsci/git-plugin/pull/1658
2. Revert the expansion of a few classes to separate Java objects https://github.com/jenkinsci/git-plugin/pull/1657
Incremental builds will be available for each of those.
Hi,
Just step in because I faced the same and needed to urgently rollback ;(
I can provide more detail on my setup but the error is the same
returned status code 128: [05:43:28.670+02:00] - stdout: [05:43:28.670+02:00] - stderr: Host key verification failed. [05:43:28.670+02:00] - fatal: Could not read from remote repository. [05:43:28.670+02:00] - [05:43:28.670+02:00] - Please make sure you have the correct access rights [05:43:28.670+02:00] - and the repository exists. [05:43:28.670+02:00] - [05:43:28.670+02:00] - at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846) [05:43:28.670+02:00] - at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185) [05:43:28.670+02:00] - at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635) [05:43:28.670+02:00] - at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
Rollbacking all plugin with exact same config works
List of plugins
active-directory:2.36
analysis-model-api:12.5.0
ansible:403.v8d0ca_dcb_b_502
ansicolor:1.0.4
ant:511.v0a_a_1a_334f41b_
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f
artifactory-artifact-manager:122.v0b_d7cb_89ca_34
artifactory-client-api:2.18.0-35.vee37a_72ccd74
asm-api:9.7-33.v4d23ef79fcc8
atlassian-bitbucket-server-integration:4.0.0
authentication-tokens:1.119.v50285141b_7e1
badge:2.1
basic-branch-build-strategies:81.v05e333931c7d
bitbucket-kubernetes-credentials:336.vc0a_911cde608
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-blocker-plugin:166.vc82fc20b_a_ed6
build-discarder:139.v05696a_7fe240
build-name-setter:2.4.3
build-user-vars-plugin:176.vb_9c7907fd524
build-with-parameters:76.v9382db_f78962
byte-buddy-api:1.15.1-59.v81e90f37c4c6
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.951.v5f91d88d76b_b_
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-math3-api:3.6.1-4.vdd4613817d74
commons-text-api:1.12.0-129.v99a_50df237f7
config-file-provider:978.v8e85886ffdc4
configuration-as-code:1850.va_a_8c31d3158b_
coverage-badges-extension:47.v41e62ecf0928
coverage:1.16.1
credentials-binding:681.vf91669a_32e45
credentials:1378.v81ef4269d764
custom-folder-icon:2.14
customizable-header:135.vf8ce4237feb_c
customize-build-now:17.ve5db_875e5343
dark-theme:479.v661b_1b_911c01
data-tables-api:2.1.6-1
database-postgresql:100.v2418e0a_c6909
database:247.v244b_d85f086d
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
docker-workflow:580.vc0c340686b_54
dotnet-sdk:1.4.0
dtkit-api:3.0.2
durable-task:577.v2a_8a_4b_7c0247
echarts-api:5.5.1-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
embeddable-build-status:487.va_0ef04c898a_2
extended-read-permission:53.v6499940139e5
extension-filter:135.v703ff01ca_817
extra-tool-installers:139.v723fee51b_7f2
file-operations:266.v9d4e1eb_235b_a_
file-parameters:339.v4b_cc83e11455
flatpickr-api:4.6.13-5.v534d8025a_a_59
flyway-api:9.22.3-151.v475c057b_07fc
flyway-runner:190.vd433a_679fa_b_8
font-awesome-api:6.6.0-2
forensics-api:2.6.0
generic-tool:1.1
git-client:5.0.0
git-forensics:2.2.1
git-parameter:0.9.19
git:5.5.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1797.v86fdb_4d57d43
github-checks:589.v845136f916cd
github:1.40.0
gitlab-api:5.6.0-97.v6603a_83f8690
gitlab-branch-source:710.v6f19df32544b_
gitlab-kubernetes-credentials:259.v7c72898df530
gradle:2.13
gson-api:2.11.0-41.v019fcf6125dc
h2-api:11.1.4.199-30.v1c64e772f3a_c
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hidden-parameter:237.v4b_df26c7a_f0e
htmlpublisher:1.36
http_request:1.19
implied-labels:342.vf0a_690315013
inline-pipeline:1.0.3
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jacoco:3.3.6
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:280.v050b_5c849f69
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jersey2-api:2.44-151.v6df377fff741
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.89
jobcacher-artifactory-storage:51.v06da_175e2655
jobcacher:551.ve0b_00cb_1b_85c
joda-time-api:2.13.0-85.vb_64d1c2921f1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit-attachments:239.v9e003a_c80a_8c
junit-sql-storage:345.v5094de0a_b_f4d
junit:1300.v03d9d8a_cf1fb_
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials-provider:1.262.v2670ef7ea_0c5
kubernetes-credentials:190.v03c305394deb_
kubernetes:4288.v1719f9d0c854
ldap:725.v3cb_b_711b_1a_ef
locale:519.v4e20f313cfa_f
lockable-resources:1310.v99ca_947ed698
login-theme:146.v64a_da_cf70ea_6
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-40.v58107308b_7a_7
markdown-formatter:220.v1a_262cd9f77f
mask-passwords:173.v6a_077a_291eb_5
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.2-125.v200281b_61d59
mina-sshd-api-core:2.13.2-125.v200281b_61d59
mstest:1.0.5
next-build-number:1.8
next-executions:327.v136ff959e97b_
nodejs:1.6.2
nunit:485.ve8a_85357320d
oidc-provider:79.v46f0066a_d813
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
openshift-client:1.1.0.424.v829cb_ccf8798
openshift-k8s-credentials:168.v79a_983191991
opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4
oss-symbols-api:71.v08d42ee3785d
pam-auth:1.11
parameter-separator:166.vd0120849b_386
parameterized-scheduler:277.v61a_4b_a_49a_c5c
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:340.v28cecee8b_25f
pipeline-groovy-lib:730.ve57b_34648c63
pipeline-input-step:495.ve9c153f6067b_
pipeline-maven-api:1421.v610fa_b_e2d60e
pipeline-maven-database:1421.v610fa_b_e2d60e
pipeline-maven:1421.v610fa_b_e2d60e
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
pipeline-npm:204.v4dc4c2202625
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
platformlabeler:2617.v5444054f5e35
plugin-util-api:4.1.0
postgresql-api:42.7.2-40.v76d376d65c77
postgresql-fingerprint-storage:274.v417e436b_1c5e
prism-api:1.29.0-17
prometheus:784.vea_eca_f6592eb_
pubsub-light:1.18
purge-build-queue-plugin:88.v23b_97b_f2c7a_d
resource-disposer:0.23
scm-api:696.v778d637b_a_762
scm-filter-branch-pr:148.v0b_5f06e8b_c84
script-security:1362.v67dc1f0e1b_b_3
sidebar-link:2.4.1
simple-theme-plugin:196.v96d9592f4efa_
skip-notifications-trait:313.vd1337c8f8134
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.17.2
ssh-agent:376.v8933585c69d3
ssh-credentials:343.v884f71d78167
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
strict-crumb-issuer:2.1.1
structs:338.v848422169819
theme-manager:262.vc57ee4a_eda_5d
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.3
variant:60.v7290fc0eb_b_cd
versioncolumn:243.vda_c20eea_a_8a_f
warnings-ng:11.6.0
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3964.v0767b_4b_a_0b_fa_
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:926.v9f4f9b_b_98c19
ws-cleanup:0.46
List of plugin that doesn't cause issue (1 week difference more less)
active-directory:2.36 analysis-model-api:12.5.0 ansible:403.v8d0ca_dcb_b_502 ansicolor:1.0.4 ant:511.v0a_a_1a_334f41b_ antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-117.v4d95117cd34f artifactory-artifact-manager:122.v0b_d7cb_89ca_34 artifactory-client-api:2.18.0-35.vee37a_72ccd74 asm-api:9.7-33.v4d23ef79fcc8 atlassian-bitbucket-server-integration:4.0.0 authentication-tokens:1.119.v50285141b_7e1 badge:2.0 basic-branch-build-strategies:81.v05e333931c7d bitbucket-kubernetes-credentials:336.vc0a_911cde608 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_ branch-api:2.1178.v969d9eb_c728e build-blocker-plugin:166.vc82fc20b_a_ed6 build-discarder:139.v05696a_7fe240 build-name-setter:2.4.3 build-user-vars-plugin:176.vb_9c7907fd524 build-with-parameters:76.v9382db_f78962 byte-buddy-api:1.15.1-59.v81e90f37c4c6 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.1 cloudbees-bitbucket-branch-source:888.v8e6d479a_1730 cloudbees-folder:6.951.v5f91d88d76b_b_ commons-compress-api:1.26.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-math3-api:3.6.1-4.vdd4613817d74 commons-text-api:1.12.0-129.v99a_50df237f7 config-file-provider:978.v8e85886ffdc4 configuration-as-code:1850.va_a_8c31d3158b_ coverage-badges-extension:47.v41e62ecf0928 coverage:1.16.1 credentials-binding:681.vf91669a_32e45 credentials:1371.vfee6b_095f0a_3 custom-folder-icon:2.14 customizable-header:135.vf8ce4237feb_c customize-build-now:17.ve5db_875e5343 dark-theme:479.v661b_1b_911c01 data-tables-api:2.1.6-1 database-postgresql:100.v2418e0a_c6909 database:247.v244b_d85f086d display-url-api:2.204.vf6fddd8a_8b_e9 docker-commons:443.v921729d5611d docker-workflow:580.vc0c340686b_54 dotnet-sdk:1.4.0 dtkit-api:3.0.2 durable-task:577.v2a_8a_4b_7c0247 echarts-api:5.5.1-1 eddsa-api:0.3.0-4.v84c6f0f4969e email-ext:1814.v404722f34263 embeddable-build-status:487.va_0ef04c898a_2 extended-read-permission:53.v6499940139e5 extension-filter:135.v703ff01ca_817 extra-tool-installers:139.v723fee51b_7f2 file-operations:266.v9d4e1eb_235b_a_ file-parameters:339.v4b_cc83e11455 flatpickr-api:4.6.13-5.v534d8025a_a_59 flyway-api:9.22.3-151.v475c057b_07fc flyway-runner:190.vd433a_679fa_b_8 font-awesome-api:6.6.0-2 forensics-api:2.5.0 generic-tool:1.1 git-client:5.0.0 git-forensics:2.2.1 git-parameter:0.9.19 git:5.4.1 github-api:1.321-468.v6a_9f5f2d5a_7e github-branch-source:1797.v86fdb_4d57d43 github-checks:589.v845136f916cd github:1.40.0 gitlab-api:5.6.0-97.v6603a_83f8690 gitlab-branch-source:710.v6f19df32544b_ gitlab-kubernetes-credentials:259.v7c72898df530 gradle:2.12.1 gson-api:2.11.0-41.v019fcf6125dc h2-api:11.1.4.199-30.v1c64e772f3a_c handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:237.v4b_df26c7a_f0e htmlpublisher:1.36 http_request:1.19 implied-labels:342.vf0a_690315013 inline-pipeline:1.0.3 instance-identity:185.v303dc7c645f9 ionicons-api:74.v93d5eb_813d5f jackson2-api:2.17.0-379.v02de8ec9f64c jacoco:3.3.6 jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:280.v050b_5c849f69 javax-activation-api:1.2.0-7 javax-mail-api:1.6.2-10 jaxb:2.3.9-1 jdk-tool:80.v8a_dee33ed6f0 jersey2-api:2.44-151.v6df377fff741 jjwt-api:0.11.5-112.ve82dfb_224b_a_d job-dsl:1.89 jobcacher-artifactory-storage:51.v06da_175e2655 jobcacher:551.ve0b_00cb_1b_85c joda-time-api:2.13.0-85.vb_64d1c2921f1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit-attachments:239.v9e003a_c80a_8c junit-sql-storage:345.v5094de0a_b_f4d junit:1300.v03d9d8a_cf1fb_ kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials-provider:1.262.v2670ef7ea_0c5 kubernetes-credentials:189.v90a_488b_d1d65 kubernetes:4287.v73451380b_576 ldap:725.v3cb_b_711b_1a_ef locale:519.v4e20f313cfa_f lockable-resources:1301.v0e3b_da_4b_4462 login-theme:146.v64a_da_cf70ea_6 mailer:472.vf7c289a_4b_420 mapdb-api:1.0.9-40.v58107308b_7a_7 markdown-formatter:220.v1a_262cd9f77f mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.2 matrix-project:832.va_66e270d2946 metrics:4.2.21-451.vd51df8df52ec mina-sshd-api-common:2.13.2-125.v200281b_61d59 mina-sshd-api-core:2.13.2-125.v200281b_61d59 mstest:1.0.5 next-build-number:1.8 next-executions:327.v136ff959e97b_ nodejs:1.6.2 nunit:485.ve8a_85357320d oidc-provider:79.v46f0066a_d813 okhttp-api:4.11.0-172.vda_da_1feeb_c6e openshift-client:1.1.0.424.v829cb_ccf8798 openshift-k8s-credentials:168.v79a_983191991 opentelemetry-api:1.40.0-36.v1e02b_b_4db_8f4 oss-symbols-api:67.v6cc456b_ed2fb_ pam-auth:1.11 parameter-separator:166.vd0120849b_386 parameterized-scheduler:277.v61a_4b_a_49a_c5c parameterized-trigger:806.vf6fff3e28c3e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-graph-view:340.v28cecee8b_25f pipeline-groovy-lib:730.ve57b_34648c63 pipeline-input-step:495.ve9c153f6067b_ pipeline-maven-api:1421.v610fa_b_e2d60e pipeline-maven-database:1421.v610fa_b_e2d60e pipeline-maven:1421.v610fa_b_e2d60e pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83 pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83 pipeline-npm:204.v4dc4c2202625 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83 pipeline-utility-steps:2.17.0 plain-credentials:183.va_de8f1dd5a_2b_ platformlabeler:2617.v5444054f5e35 plugin-util-api:4.1.0 postgresql-api:42.7.2-40.v76d376d65c77 postgresql-fingerprint-storage:274.v417e436b_1c5e prism-api:1.29.0-17 prometheus:784.vea_eca_f6592eb_ pubsub-light:1.18 purge-build-queue-plugin:88.v23b_97b_f2c7a_d resource-disposer:0.23 scm-api:696.v778d637b_a_762 scm-filter-branch-pr:148.v0b_5f06e8b_c84 script-security:1362.v67dc1f0e1b_b_3 sidebar-link:2.4.1 simple-theme-plugin:196.v96d9592f4efa_ skip-notifications-trait:313.vd1337c8f8134 snakeyaml-api:2.3-123.v13484c65210a_ sonar:2.17.2 ssh-agent:376.v8933585c69d3 ssh-credentials:343.v884f71d78167 ssh-slaves:2.973.v0fa_8c0dea_f9f sshd:3.330.vc866a_8389b_58 strict-crumb-issuer:2.1.1 structs:338.v848422169819 theme-manager:262.vc57ee4a_eda_5d timestamper:1.27 token-macro:400.v35420b_922dcb_ trilead-api:2.147.vb_73cc728a_32e uno-choice:2.8.3 variant:60.v7290fc0eb_b_cd versioncolumn:243.vda_c20eea_a_8a_f warnings-ng:11.6.0 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1336.vee415d95c521 workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3961.ve48ee2c44a_b_3 workflow-durable-task-step:1371.vb_7cec8f3b_95e workflow-job:1436.vfa_244484591f workflow-multibranch:795.ve0cb_1f45ca_9a_ workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:678.v3ee58b_469476 workflow-support:920.v59f71ce16f04 ws-cleanup:0.46
The error doesn't seem to arrive in all condition
- Checking out global libraries on the controller seems to work
- Explicit git checkout in a scripted pipeline seems to work
- Implicit git checkout in a declararive pipeline doesn't work in a multibranch
Hope it can help
I believe this might be related specifically to Bitbucket multibranch pipelines? I've tried the exact same pipeline with a GitHub multibranch, a Git multibranch and a Bitbucket multibranch pipeline and only the Bitbucket pipeline fails!
Affected uses, can you report your source server type and job setup?
paybas what is the host key configuration setting that you are using? Is it manually provided keys, known hosts file, or accept new?
markewaite we are using "Git Host Key Verification Configuration" = Manually provided keys
yoerg I downgraded pretty quickly but looking back at the failed jobs it does indeed seem to suggest that regular pipelines were unaffected but Bitbucket multibranch pipelines which use https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ were affected.
I have no non-Bitbucket multibranch pipelines to compare with however.
Same here. Multibranch bitbucket pipeline seems only affected. Not regular ones
Looks like the BitBucket multibranch plugin overrides the
auth and with "reordering of extension additions" the Creds from the Bitbucket multibranch plugin gets active and not the one from the git plugin.
Tried it:
PR-1657 Host key verification failed.
PR-1658 Verifying host key using manually-configured host key entries
Can confirm this.
Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?
I found another workaround beside downgrading the plugin.
1.)
In case you use the Bitbucket multibranch setup, do not checkout via ssh anymore, use the PAT token instead.
With the ssh checkout, we had the effect that all pull request jobs directly fail. Non pull request jobs were somehow not affected. Without the ssh checkout, the pull request jobs start again.
2.) In case that you invoke the checkout step directly in your Jenkinsfile, consider the following.
Try to explicitly inject the credentials instead of relying on the scm variable.
So, instead of:
gitSCM = checkout(
[
$class : 'GitSCM',
branches : scm.branches,
doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
userRemoteConfigs : scm.userRemoteConfigs,
extensions : [
[$class : "CloneOption",
noTags : noTags,
shallow : isShallowClone,
depth : cloneDepth,
reference: ''],
],
],
)
Use something like:
gitSCM = checkout(
[
$class : 'GitSCM',
branches : scm.branches,
doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
userRemoteConfigs : [[url: scm.userRemoteConfigs[0].url, credentialsId: "YourCredentialsID",
extensions : [
[$class : "CloneOption",
noTags : noTags,
shallow : isShallowClone,
depth : cloneDepth,
reference: ''],
],
],
)
Maybe this is helpful for somebody.
Should a fix be on bitbucket-branch-source-plugin then instead of rollback the PR ?
Thanks for the suggestion. I'm releasing git plugin 5.5.1 that removes the pull request so that I then have time to implement the change with appropriate test environment improvements that will detect this type of failure. I believe that my test environment already has the ability to show this failure, but I didn't check the status of every test job after making that change. I knew the change was risky and had evaluated it in several different ways (my environment, plugin tests, plugin BOM, acceptance test harness), but did not specifically check the behavior of Bitbucket branch source plugin multibranch Pipelines with ssh authentication. I believe that I have those job types, but I did not check their status.
I've released git plugin 5.5.1 that reverts the reordering of extension additions. My sincere apologies to everyone affected by the issue.
I'll work on additional automation that detects that type of failure during pull request evaluation, since the additional automation will help many other pull requests as well.
My sincere apologies to everyone affected by the issue.
I don't see why you should apologize. This is an open-source project maintained largely by volunteers and we're all using it for free.
Besides: the ecosystem is massive, so every possible plugin interaction is impossible to test beforehand.
We should be thanking you for your dedication and rapid response.
It's not just host key verification, actually testing this the GitSCM was using the Pipeline Multibranch scan credentials instead of the SSH Credentials (that could check in the build log that is not showing using GIT_SSH anymore. Bitbucket Branch source in the scenario of SSHCheckoutTrait configures a GitSCM with the SSH credentials but also an authenticator extension (supposed to decorate the GitClient with inferred authentication from the Pipeline Multibranch scan credentials). Which is probably wrong.
The way authentication is pre-configures is something I was trying to address as part of https://issues.jenkins.io/browse/JENKINS-73471. I have added a test for this scenario in Bitbucket Branch Source to cover this https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/867/files. Hopefully this covers it for the next attempt to apply JENKINS-73677
Should further be prevented by bitbucket-branch-source-plugin:906.vedf430cb_4481.
Same here, with OpenStack cloud agent and "Manually provided keys" strategy. Regression in 5.5.0, downgrade to 5.4.1 fixes the issue.