Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58967

Credentials not available after upgrade to LTS 2.176.2

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Blocker Blocker

      In a Multibranch pipeline we're using the withCredentials syntax in several places:

      withCredentials([usernamePassword(credentialsId: 'name-of-credentials', passwordVariable: 'PASSWORD', usernameVariable: 'USER')]) {
       // some block
      }
      

       

      This has worked fine for several weeks but since the upgrade we get:

      [Pipeline] withCredentials
      [Pipeline] // withCredentials
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] echo
      *15:07:06*  failed
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      [BFA] Scanning build for known causes...
      [BFA] No failure causes found
      [BFA] Done. 0s
      ERROR: Could not find credentials entry with ID 'name-of-credentials'
      Finished: FAILURE
      

      The pipeline has not changed, the credentials have not been updated.

      I have found that the snippet generator displays an empty drop down list when I start it from within the "trunk" directory of the Multibranch pipeline.

      At one point I added a new credential and limited it to the folder of the Multibranch pipeline.
      When I start the snippet generator in that directory the new credential as well as the old credentials are both visible.
      When I start the snippet generator at the top level only the old credential is visible (as expected).

      The problem is that within the branches (currently only trunk) of the multibranch pipeline the credentials are not visible.

          [JENKINS-58967] Credentials not available after upgrade to LTS 2.176.2

          Bram Mertens added a comment -

          As a workaround I tried to setup a regular jenkins pipeline job using the same jenkinsfile.

          This fails with the same error:

          [BFA] Done. 0s
          ERROR: Could not find credentials entry with ID 'customer-testmatch-creds'
          Finished: FAILURE

          The jenkins pipeline looks like:

                    withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds', passwordVariable: 'TM_PASSWORD', usernameVariable: 'TM_USER')]) {
                    ...
                    }

          If I use the Snippet Generator at this level the dropdown is correctly populated with both the existing and the new credentials.

          The snippet that is generated looks ok:

          withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds', passwordVariable: 'TM_PASSWORD', usernameVariable: 'TM_USER')]) {
              // some block
          }
          

          I cannot find any errors in the jenkins log file.

          How can I troubleshoot this further? This is blocking our regression tests and delivery to our customer.

          Bram Mertens added a comment - As a workaround I tried to setup a regular jenkins pipeline job using the same jenkinsfile. This fails with the same error: [BFA] Done. 0s ERROR: Could not find credentials entry with ID 'customer-testmatch-creds' Finished: FAILURE The jenkins pipeline looks like: withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds' , passwordVariable: 'TM_PASSWORD' , usernameVariable: 'TM_USER' )]) { ... } If I use the Snippet Generator at this level the dropdown is correctly populated with both the existing and the new credentials. The snippet that is generated looks ok: withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds' , passwordVariable: 'TM_PASSWORD' , usernameVariable: 'TM_USER' )]) { // some block } I cannot find any errors in the jenkins log file. How can I troubleshoot this further? This is blocking our regression tests and delivery to our customer.

          Bram Mertens added a comment -

          I can reproduce the problem with a minimal new pipeline:

          node('serverX&&windows') {
              withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds2', passwordVariable: 'TM_PASSWORD', usernameVariable: 'TM_USER')]) {
              echo 'running with new credentials'
              }
          }
          

          The credentials with ID customer-testmatch-creds2 was created after the upgrade.

          Bram Mertens added a comment - I can reproduce the problem with a minimal new pipeline: node( 'serverX&&windows' ) { withCredentials([usernamePassword(credentialsId: 'customer-testmatch-creds2' , passwordVariable: 'TM_PASSWORD' , usernameVariable: 'TM_USER' )]) { echo 'running with new credentials' } } The credentials with ID customer-testmatch-creds2 was created after the upgrade.

          Bram Mertens added a comment -

          Using credentials works fine on a test jenkins setup:

          RHEL 7.6 version 2.179

          credentials plugin 2.2.0

          credentials binding plugin 1.19

          Pipeline plugin 2.6

          Pipeline: Multibranch plugin 2.21

           

          Also works fine on another older installation:

          Jenkins 2.143 on RHEL 6.7

          credentials plugin 2.1.18

          crdentials binding plugin 1.16

          pipeline: multibranch: 2.20 (not used in this setup)

           

          Bram Mertens added a comment - Using credentials works fine on a test jenkins setup: RHEL 7.6 version 2.179 credentials plugin 2.2.0 credentials binding plugin 1.19 Pipeline plugin 2.6 Pipeline: Multibranch plugin 2.21   Also works fine on another older installation: Jenkins 2.143 on RHEL 6.7 credentials plugin 2.1.18 crdentials binding plugin 1.16 pipeline: multibranch: 2.20 (not used in this setup)  

          Bram Mertens added a comment -

          The attached stacktrace was generated by running the test pipeline (not multibranch) after downgrading the credentials binding plugin to 1.13.

          Bram Mertens added a comment - The attached stacktrace was generated by running the test pipeline (not multibranch) after downgrading the credentials binding plugin to 1.13.

          Bram Mertens added a comment -

          The credentials test pipeline I created also works on another instance of the same LTS version.

          RHEL 7.6 , jenkins-2.176.2-1.1.noarch, 

          Several plugins are slightly older. The ones that catch my eye are:

          credentials 2.2.0

          credentials-binding 1.19 

           

          Versus on the jenkins master that fails:

          credentials 2.2.1

          credentials-binding 1.20

           

          Bram Mertens added a comment - The credentials test pipeline I created also works on another instance of the same LTS version. RHEL 7.6 , jenkins-2.176.2-1.1.noarch,  Several plugins are slightly older. The ones that catch my eye are: credentials 2.2.0 credentials-binding 1.19    Versus on the jenkins master that fails: credentials 2.2.1 credentials-binding 1.20  

          Bram Mertens added a comment -

          I tried to dowgrade the credentials and credentials binding plugins to credentials 2.2.0 and credentials binding 1.19 and even created a new credentials entry but the problem persists.

          Bram Mertens added a comment - I tried to dowgrade the credentials and credentials binding plugins to credentials 2.2.0 and credentials binding 1.19 and even created a new credentials entry but the problem persists.

          Bram Mertens added a comment -

          A colleague found out that the problem is caused by the fact that as part of the upgrade the build authorization was configured to run as a specific user.

          The user that is configured does not have the Jobs/Configure permission.

          If we grant that permission the job is able to get the credentials as before.

          We will investigate the different options described in the Credentials API user guide to avoid granting that user Jobs/Configure permission.

          Bram Mertens added a comment - A colleague found out that the problem is caused by the fact that as part of the upgrade the build authorization was configured to run as a specific user. The user that is configured does not have the Jobs/Configure permission. If we grant that permission the job is able to get the credentials as before. We will investigate the different options described in the Credentials API user guide to avoid granting that user Jobs/Configure permission.

          This seems to be working:

          • Add a "jenkins-build" user to the security realm.
          • Configure Authorize Project to run all builds as the "jenkins-build" user, without allowing per-job configuration.
          • Add -Dcom.cloudbees.plugins.credentials.UseItemPermission=true to the Java options of the Jenkins master.
          • Configure Role-based Authorization Strategy like this:
            • Define the global role "build" with only these global permissions:
              • Overall/Read (might not be necessary)
              • Credentials/UseItem (this requires the option that was set above)
              • Agent/Build
              • Job/Read
            • Assign the global role "build" to the user "jenkins-build".
          • Add a certificate credential to the global credential domain of a multibranch pipeline job.
          • Reference the credential using withCredentials in the Jenkinsfiles of branches of that job.

          If I then log in as the "jenkins-build" user, I do not see the credentials, because of the missing Credentials/View permission. However, the builds can use the credentials just fine.

          Kalle Niemitalo added a comment - This seems to be working: Add a "jenkins-build" user to the security realm. Configure Authorize Project to run all builds as the "jenkins-build" user, without allowing per-job configuration. Add -Dcom.cloudbees.plugins.credentials.UseItemPermission=true to the Java options of the Jenkins master. Configure Role-based Authorization Strategy like this: Define the global role "build" with only these global permissions: Overall/Read (might not be necessary) Credentials/UseItem (this requires the option that was set above) Agent/Build Job/Read Assign the global role "build" to the user "jenkins-build". Add a certificate credential to the global credential domain of a multibranch pipeline job. Reference the credential using withCredentials in the Jenkinsfiles of branches of that job. If I then log in as the "jenkins-build" user, I do not see the credentials, because of the missing Credentials/View permission. However, the builds can use the credentials just fine.

          Matt Sicker added a comment -

          Sounds like this wasn't a regression then?

          Matt Sicker added a comment - Sounds like this wasn't a regression then?

            Unassigned Unassigned
            bram_mertens Bram Mertens
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: