Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70275

Folder-level credentials unavailable to Multibranch pipelines whenever Jenkins restarts

    • 3.4.1

      After configuring Jenkins and Bitbucket to trigger building a Multibranch Pipeline Job in Jenkins after a pull request is created in Bitbucket, initially, everything works OK.

      However, after a couple of days, creating a PR in Bitbucket suddenly stopped triggering builds. This is what the Multibranch Pipeline Events log shows after a successful trigger:

      Successful trigger
      [Wed Dec 14 09:14:52 CET 2022] Received com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadPullRequestEvent CREATED event from example-trigger with timestamp Wed Dec 14 09:14:52 CET 2022
       > /usr/bin/git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-dfb7ecede3df9ca94f084a3891c26a05/.git # timeout=10
      Setting origin to https://example.com/example-trigger.git
       > /usr/bin/git config remote.origin.url https://example.com/example-trigger.git # timeout=10
      Fetching & pruning origin...
      Listing remote references...
       > /usr/bin/git config --get remote.origin.url # timeout=10
       > /usr/bin/git --version # timeout=10
       > git --version # 'git version 1.8.3.1'
      using GIT_ASKPASS to set credentials Example credentials
       > /usr/bin/git ls-remote -h https://example.com/example-trigger.git # timeout=10
      Fetching upstream changes from origin
       > /usr/bin/git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials Example credentials
       > /usr/bin/git fetch --tags --progress --prune origin +refs/heads/*:refs/remotes/origin/* # timeout=10
      Checking branches...
        Checking branch example-branch/main.py-1671005685967
            'jenkins/Jenkinsfile' found
          Met criteria
      Scheduled build for branch: example-branch/main.py-1671005685967
      Processed 4 branches (query complete)
      [Wed Dec 14 09:14:56 CET 2022] com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadPullRequestEvent CREATED event from example-trigger with timestamp Wed Dec 14 09:14:52 CET 2022 processed in 3.3 sec

      And here's what a failed trigger looks like:

      Failed trigger
      [Wed Dec 14 09:13:27 CET 2022] Received com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadPullRequestEvent CREATED event from example-trigger with timestamp Wed Dec 14 09:13:27 CET 2022
       > /usr/bin/git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-dfb7ecece3dc9ca94f084a3891c26a05/.git # timeout=10
      Setting origin to https://example.com/example-trigger.git
       > /usr/bin/git config remote.origin.url https://example.com/example-trigger.git # timeout=10
      Fetching & pruning origin...
      Listing remote references...
       > /usr/bin/git config --get remote.origin.url # timeout=10
       > /usr/bin/git --version # timeout=10
       > git --version # 'git version 1.8.3.1'
       > /usr/bin/git ls-remote -h https://example.com/example-trigger.git # timeout=10
      [Wed Dec 14 09:13:27 CET 2022] com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadPullRequestEvent CREATED event from example-trigger with timestamp Wed Dec 14 09:13:27 CET 2022 processed in 0.26 sec

      It appears it skips setting credentials for failed triggers. As a workaround, if you configure the Multibranch Pipeline Job and click "Apply" without changing any settings, everything works as intended again.

      This issue seems to be occurring after upgrading Jenkins and restarting. Could it be caused by the use of transients?

          [JENKINS-70275] Folder-level credentials unavailable to Multibranch pipelines whenever Jenkins restarts

          Thanks for the report Cas. This will be triaged soon but in the meantime I've raised the priority to Major as although the fix is simple, it would be an arduous problem on bigger instances.

          Martin Henschke added a comment - Thanks for the report Cas. This will be triaged soon but in the meantime I've raised the priority to Major as although the fix is simple, it would be an arduous problem on bigger instances.

          Cas Dekkers added a comment -

          This issue seems to be occurring after upgrading Jenkins and restarting.

          mhenschke_atlassian thanks for your reply. To clarify, this issue has been occurring for a while each time after upgrading/restarting, so I doubt the cause will be specifically related to the latest version of Jenkins.

          Cas Dekkers added a comment - This issue seems to be occurring after upgrading Jenkins and restarting. mhenschke_atlassian thanks for your reply. To clarify, this issue has been occurring for a while each time after upgrading/restarting, so I doubt the cause will be specifically related to the latest version of Jenkins.

          Barry Qiao added a comment -

          Hi mhenschke_atlassian ,

          We are having the similar issue that reports 'Authentication failed' every time for either 'Multibranch Scan' or 'Pipeline' after jenkins service restart . But our Jenkins server is in a low version 3.346.1.

          As further verification, it seems only happens with the credential created in 'Folder' level. Does not happen with the ones in 'Global' level.

          Tried to downgrade the Bitbucket Server Integration to 3.1.0 , the error turned into 'java.lang.NullPointerException'. 

          Could you please explain more details about this issue?  and kindly suggest any workaround for this. 

           

           Thanks.

           

          Barry Qiao added a comment - Hi mhenschke_atlassian  , We are having the similar issue that reports 'Authentication failed' every time for either 'Multibranch Scan' or 'Pipeline' after jenkins service restart . But our Jenkins server is in a low version 3.346.1. As further verification, it seems only happens with the credential created in 'Folder' level. Does not happen with the ones in 'Global' level. Tried to downgrade the Bitbucket Server Integration to 3.1.0 , the error turned into 'java.lang.NullPointerException'.  Could you please explain more details about this issue?  and kindly suggest any workaround for this.     Thanks.  

          Cas Dekkers added a comment - - edited

          As further verification, it seems only happens with the credential created in 'Folder' level. Does not happen with the ones in 'Global' level.

          mhenschke_atlassian barryqiao I, too, can confirm that, for our instance, this is a problem that relates only to credentials on the folder level (and not the global level).

          Cas Dekkers added a comment - - edited As further verification, it seems only happens with the credential created in 'Folder' level. Does not happen with the ones in 'Global' level. mhenschke_atlassian barryqiao I, too, can confirm that, for our instance, this is a problem that relates only to credentials on the folder level (and not the global level).

          Cas Dekkers added a comment -

          Any updates?

          Cas Dekkers added a comment - Any updates?

          Barry Qiao added a comment -

          Any updates or suggestions ?

          Barry Qiao added a comment - Any updates or suggestions ?

          Martin Henschke added a comment - - edited

          Hi Cas and Barry, thank you for your reports and apologies for the long delay on response.

          We have encountered a similar issue before, and it's related to the method in which we wrap the Git SCM source. Folder-level credentials are failing because they require the containing folder as context to retrieve (root credentials are accessible without context). We had this issue previously with folder credentials not working at all, which was resolved by adding the owner on SCMSource initialization. This bug is present because when jenkins is restarted, the SCMSource is not initialized, it is deserialized, and the config does not serialize the owner, so it behaves as if there is no context- so only global credentials will work.

          I've been able to replicate the problem and believe I have a solution, which will go into review soon- I'll also update the name of the ticket to be more descriptive of the circumstances in which this bug occurs. Sorry again for the delay.

          Martin Henschke added a comment - - edited Hi Cas and Barry, thank you for your reports and apologies for the long delay on response. We have encountered a similar issue before, and it's related to the method in which we wrap the Git SCM source. Folder-level credentials are failing because they require the containing folder as context to retrieve (root credentials are accessible without context). We had this issue previously with folder credentials not working at all, which was resolved by adding the owner on SCMSource initialization. This bug is present because when jenkins is restarted, the SCMSource is not initialized, it is  deserialized , and the config does not serialize the owner, so it behaves as if there is no context- so only global credentials will work. I've been able to replicate the problem and believe I have a solution, which will go into review soon- I'll also update the name of the ticket to be more descriptive of the circumstances in which this bug occurs. Sorry again for the delay.

          Cas Dekkers added a comment -

          mhenschke_atlassian thank you for your previous response. The issue appears to have been untouched for a while now. Are there any updates? Thank you.

          N.B. I have linked the related PR: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/pull/374

          Cas Dekkers added a comment - mhenschke_atlassian thank you for your previous response. The issue appears to have been untouched for a while now. Are there any updates? Thank you. N.B. I have linked the related PR: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/pull/374

            mhenschke_atlassian Martin Henschke
            cdekkers Cas Dekkers
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: