-
Bug
-
Resolution: Not A Defect
-
Blocker
-
We recently performed a long overdue upgrade from 2.95 to LTS 2.176.2.
Jenkins master runs on RHEL 7.7 64 bit, the node where the pipeline is failing is a Windows Server 2012 R2 (64 bit).
Master is running:
java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
Node is running:
C:\>java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
Credentials binding plugin is at 1.20
Credentials plugin is at 2.2.1
Pipeline: Multibranch is at 2.21
Jenkins is running directly without reverse proxy, installed via rpm from "jenkins-stable" (http://pkg.jenkins.io/redhat-stable)
We're using chrome as a browser but the issue occurs on builds that are triggered via SCM polling so the browser is probably not relevant.We recently performed a long overdue upgrade from 2.95 to LTS 2.176.2. Jenkins master runs on RHEL 7.7 64 bit, the node where the pipeline is failing is a Windows Server 2012 R2 (64 bit). Master is running: java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode) Node is running: C:\>java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) Credentials binding plugin is at 1.20 Credentials plugin is at 2.2.1 Pipeline: Multibranch is at 2.21 Jenkins is running directly without reverse proxy, installed via rpm from "jenkins-stable" ( http://pkg.jenkins.io/redhat-stable ) We're using chrome as a browser but the issue occurs on builds that are triggered via SCM polling so the browser is probably not relevant.
In a Multibranch pipeline we're using the withCredentials syntax in several places:
withCredentials([usernamePassword(credentialsId: 'name-of-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USER')]) {
// some block
}
This has worked fine for several weeks but since the upgrade we get:
[Pipeline] withCredentials [Pipeline] // withCredentials [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // stage [Pipeline] echo *15:07:06* failed [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s ERROR: Could not find credentials entry with ID 'name-of-credentials' Finished: FAILURE
The pipeline has not changed, the credentials have not been updated.
I have found that the snippet generator displays an empty drop down list when I start it from within the "trunk" directory of the Multibranch pipeline.
At one point I added a new credential and limited it to the folder of the Multibranch pipeline.
When I start the snippet generator in that directory the new credential as well as the old credentials are both visible.
When I start the snippet generator at the top level only the old credential is visible (as expected).
The problem is that within the branches (currently only trunk) of the multibranch pipeline the credentials are not visible.
[JENKINS-58967] Credentials not available after upgrade to LTS 2.176.2
Bram Mertens
added a comment - I can reproduce the problem with a minimal new pipeline:
node('serverX&&windows') { withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds2', passwordVariable: 'TM_PASSWORD', usernameVariable: 'TM_USER')]) { echo 'running with new credentials' } }
The credentials with ID customer-testmatch-creds2 was created after the upgrade.
Bram Mertens
added a comment - I can reproduce the problem with a minimal new pipeline:
node( 'serverX&&windows' ) {
withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds2' , passwordVariable: 'TM_PASSWORD' , usernameVariable: 'TM_USER' )]) {
echo 'running with new credentials'
}
}
The credentials with ID customer-testmatch-creds2 was created after the upgrade. Bram Mertens
added a comment - Using credentials works fine on a test jenkins setup:
RHEL 7.6 version 2.179
credentials plugin 2.2.0
credentials binding plugin 1.19
Pipeline plugin 2.6
Pipeline: Multibranch plugin 2.21
Also works fine on another older installation:
Jenkins 2.143 on RHEL 6.7
credentials plugin 2.1.18
crdentials binding plugin 1.16
pipeline: multibranch: 2.20 (not used in this setup)
Bram Mertens
added a comment - Using credentials works fine on a test jenkins setup:
RHEL 7.6 version 2.179
credentials plugin 2.2.0
credentials binding plugin 1.19
Pipeline plugin 2.6
Pipeline: Multibranch plugin 2.21
Also works fine on another older installation:
Jenkins 2.143 on RHEL 6.7
credentials plugin 2.1.18
crdentials binding plugin 1.16
pipeline: multibranch: 2.20 (not used in this setup)
Bram Mertens
added a comment - The attached stacktrace was generated by running the test pipeline (not multibranch) after downgrading the credentials binding plugin to 1.13.
Bram Mertens
added a comment - The attached stacktrace was generated by running the test pipeline (not multibranch) after downgrading the credentials binding plugin to 1.13. Bram Mertens
added a comment - The credentials test pipeline I created also works on another instance of the same LTS version.
RHEL 7.6 , jenkins-2.176.2-1.1.noarch,
Several plugins are slightly older. The ones that catch my eye are:
credentials 2.2.0
credentials-binding 1.19
Versus on the jenkins master that fails:
credentials 2.2.1
credentials-binding 1.20
Bram Mertens
added a comment - The credentials test pipeline I created also works on another instance of the same LTS version.
RHEL 7.6 , jenkins-2.176.2-1.1.noarch,
Several plugins are slightly older. The ones that catch my eye are:
credentials 2.2.0
credentials-binding 1.19
Versus on the jenkins master that fails:
credentials 2.2.1
credentials-binding 1.20
Bram Mertens
added a comment - I tried to dowgrade the credentials and credentials binding plugins to credentials 2.2.0 and credentials binding 1.19 and even created a new credentials entry but the problem persists.
Bram Mertens
added a comment - I tried to dowgrade the credentials and credentials binding plugins to credentials 2.2.0 and credentials binding 1.19 and even created a new credentials entry but the problem persists. Bram Mertens
added a comment - A colleague found out that the problem is caused by the fact that as part of the upgrade the build authorization was configured to run as a specific user.
The user that is configured does not have the Jobs/Configure permission.
If we grant that permission the job is able to get the credentials as before.
We will investigate the different options described in the Credentials API user guide to avoid granting that user Jobs/Configure permission.
Bram Mertens
added a comment - A colleague found out that the problem is caused by the fact that as part of the upgrade the build authorization was configured to run as a specific user.
The user that is configured does not have the Jobs/Configure permission.
If we grant that permission the job is able to get the credentials as before.
We will investigate the different options described in the Credentials API user guide to avoid granting that user Jobs/Configure permission. 
Kalle Niemitalo
added a comment - This seems to be working:
- Add a "jenkins-build" user to the security realm.
- Configure Authorize Project to run all builds as the "jenkins-build" user, without allowing per-job configuration.
- Add -Dcom.cloudbees.plugins.credentials.UseItemPermission=true to the Java options of the Jenkins master.
- Configure Role-based Authorization Strategy like this:
- Define the global role "build" with only these global permissions:
- Overall/Read (might not be necessary)
- Credentials/UseItem (this requires the option that was set above)
- Agent/Build
- Job/Read
- Assign the global role "build" to the user "jenkins-build".
- Define the global role "build" with only these global permissions:
- Add a certificate credential to the global credential domain of a multibranch pipeline job.
- Reference the credential using withCredentials in the Jenkinsfiles of branches of that job.
If I then log in as the "jenkins-build" user, I do not see the credentials, because of the missing Credentials/View permission. However, the builds can use the credentials just fine.
Kalle Niemitalo
added a comment - This seems to be working:
Add a "jenkins-build" user to the security realm.
Configure Authorize Project to run all builds as the "jenkins-build" user, without allowing per-job configuration.
Add -Dcom.cloudbees.plugins.credentials.UseItemPermission=true to the Java options of the Jenkins master.
Configure Role-based Authorization Strategy like this:
Define the global role "build" with only these global permissions:
Overall/Read (might not be necessary)
Credentials/UseItem (this requires the option that was set above)
Agent/Build
Job/Read
Assign the global role "build" to the user "jenkins-build".
Add a certificate credential to the global credential domain of a multibranch pipeline job.
Reference the credential using withCredentials in the Jenkinsfiles of branches of that job.
If I then log in as the "jenkins-build" user, I do not see the credentials, because of the missing Credentials/View permission. However, the builds can use the credentials just fine. 
As a workaround I tried to setup a regular jenkins pipeline job using the same jenkinsfile.
This fails with the same error:
The jenkins pipeline looks like:
If I use the Snippet Generator at this level the dropdown is correctly populated with both the existing and the new credentials.
The snippet that is generated looks ok:
I cannot find any errors in the jenkins log file.
How can I troubleshoot this further? This is blocking our regression tests and delivery to our customer.