• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • Bitbucket Version: 8.16.1
      Jenkins Version: 2.44.0
      Bitbucket Branch Source Plugin Version: 856.v04c46c86f911
      Bitbucket Server Integration Plugin: 4.0.0

      I have had a number of pipelines that have been working successfully for years.

      Recently, multi-branch pipeline scans consistently fail with a message:

      The BitbucketSCMSource has been incorrectly configured, and cannot perform a retrieve. Check the configuration before running this job again.

      Going into the Jenkins pipeline configuration and simply saving the config with the exact same settings fixes the pipeline, and it starts automatically picking up builds again.

      However, after a couple of days of no builds, it enters the original state, giving the above error when scanning the pipeline, and the only way to get it starting again is to open and save the configuration.

       

          [JENKINS-72765] Bitbucket SCM Configuration unstable

          Paul added a comment -

          Hi

          Maybe update  Bitbucket Branch Source Plugin Version: 856.v04c46c86f911 to 876.v857269a_5f439 help fix this?

          Paul added a comment - Hi Maybe update  Bitbucket Branch Source Plugin Version: 856.v04c46c86f911 to 876.v857269a_5f439 help fix this?

          Paul added a comment -

          after restart instance this bug is reproduced in multibranch jobs

          Paul added a comment - after restart instance this bug is reproduced in multibranch jobs

          Hi Nick, thanks for your report.

          I need a bit more information to be able to look at this bug report. Specifically:

          • Were any changes made to your job configuration, or related configuration (e.g. credentials) between when your pipeline was working and when it was not?
          • Could you share the full pipeline scan logs before and after the save?
          • Can you share the job configuration before and after you re-save the configuration and the issue resolves? Specifically if you could share the config.xml, scrubbed of any identifying data (though if this data changes between saves, could you potentially note that)
          • You also mention "a couple of days of no builds"- was it failing to pick up webhooks that were being fired?

          Thanks,

          Martin

          Martin Henschke added a comment - Hi Nick, thanks for your report. I need a bit more information to be able to look at this bug report. Specifically: Were any changes made to your job configuration, or related configuration (e.g. credentials) between when your pipeline was working and when it was not? Could you share the full pipeline scan logs before and after the save? Can you share the job configuration before and after you re-save the configuration and the issue resolves? Specifically if you could share the config.xml, scrubbed of any identifying data (though if this data changes between saves, could you potentially note that) You also mention "a couple of days of no builds"- was it failing to pick up webhooks that were being fired? Thanks, Martin

          Caleb added a comment -

          I am seeing the same behavior in my Jenkins environment where I make a multibranch job, its working just fine, but then when I perform a Jenkins upgrade then I get into this state where I need to save the config before the multibranch job works again. Taking a look at my config.xml for the job before and after the upgrade I see this:

          % diff before_upgrade.xml after_upgrade.xml 
          55c55
          < <cloneUrl>....../t42/common.git</cloneUrl>
          ---
          > <cloneUrl/>
          59c59
          < <projectKey>T42</projectKey>
          ---
          > <projectKey>Test42</projectKey>
          66c66
          < <selfLink>...../T42/repos/common</selfLink>
          ---
          > <selfLink/> 

          Of those I think the problem is the projectKey if I had to guess, it seems like its setting the projectKey to the value of the projectName which is causing the problems.

          Caleb added a comment - I am seeing the same behavior in my Jenkins environment where I make a multibranch job, its working just fine, but then when I perform a Jenkins upgrade then I get into this state where I need to save the config before the multibranch job works again. Taking a look at my config.xml for the job before and after the upgrade I see this: % diff before_upgrade.xml after_upgrade.xml 55c55 < <cloneUrl>....../t42/common.git</cloneUrl> --- > <cloneUrl/> 59c59 < <projectKey>T42</projectKey> --- > <projectKey>Test42</projectKey> 66c66 < <selfLink>...../T42/repos/common</selfLink> --- > <selfLink/> Of those I think the problem is the projectKey if I had to guess, it seems like its setting the projectKey to the value of the projectName  which is causing the problems.

          Nick added a comment - - edited

          This may make sense as I have a cron job that checks for updates and automatically applies them nightly.

          mhenschke_atlassian, it fails to respond to webhooks being fired with the message in my report in the scan pipeline web hooks.  Next time it happens, I’ll try to add more details.

          Nick added a comment - - edited This may make sense as I have a cron job that checks for updates and automatically applies them nightly. mhenschke_atlassian , it fails to respond to webhooks being fired with the message in my report in the scan pipeline web hooks.  Next time it happens, I’ll try to add more details.

          Scott added a comment -

          This happened to me as well. Bitbucket Branch Source Plugin Version: 883.v041fa_695e9c2, Bitbucket Server Integration: 4.0.0

          If this continues then I will have to stop using Bitbucket Server Integration for multibranch jobs.

          Scott added a comment - This happened to me as well. Bitbucket Branch Source Plugin Version: 883.v041fa_695e9c2, Bitbucket Server Integration: 4.0.0 If this continues then I will have to stop using Bitbucket Server Integration for multibranch jobs.

          Scott added a comment -

          Adding some more light to the issue for anyone else who might encounter it, and to help debug. Here is the rough order of things in our case. Note that this played out over the course of several months:

          1. Installed the Bitbucket Server Integration plugin version 4.0.0
          2. Restarted jenkins
          3. Updated jobs from using Bitbucket Branch Source Plugin to Bitbucket Server Integration
          4. Job ran fine
          5. Restarted jenkins
            1. Jobs started entering this state after the restart

          As mentioned in other comments, there was never a change to the config.xml file for the broken job. I did however discover a state.xml file, whose contents appeared to have been invalided with the restart.

          The mitigation is to update the config.xml, either through the UI or jobdsl. A simple white space change to the job description is sufficient. Once the update was made I noticed the state.xml file would change. Below is one example. In this example, you see the projectKey was updated from the lowercase version to the technically-correct upper case version:

          diff state.xml state-old.xml 
          11,12c11,12
          <             <projectKey>FOO</projectKey>
          <             <projectName>Foo</projectName>
          ---
          >             <projectKey>foo</projectKey>
          >             <projectName>foo</projectName> 

          My remaining questions are:

          • Why did the restart in step 5 cause this?
          • Will the jobs ever regress and break again?

          Scott added a comment - Adding some more light to the issue for anyone else who might encounter it, and to help debug. Here is the rough order of things in our case. Note that this played out over the course of several months: Installed the Bitbucket Server Integration plugin version 4.0.0 Restarted jenkins Updated jobs from using Bitbucket Branch Source Plugin to Bitbucket Server Integration Job ran fine Restarted jenkins Jobs started entering this state after the restart As mentioned in other comments, there was never a change to the config.xml file for the broken job. I did however discover a state.xml file, whose contents appeared to have been invalided with the restart. The mitigation is to update the config.xml, either through the UI or jobdsl. A simple white space change to the job description is sufficient. Once the update was made I noticed the state.xml file would change. Below is one example. In this example, you see the projectKey was updated from the lowercase version to the technically-correct upper case version: diff state.xml state-old.xml 11,12c11,12 < <projectKey>FOO</projectKey> < <projectName>Foo</projectName> --- > <projectKey>foo</projectKey> > <projectName>foo</projectName> My remaining questions are: Why did the restart in step 5 cause this? Will the jobs ever regress and break again?

          Scott added a comment -

          After weeks of toil, I think I finally pinpointed the problem!

          The problem is that if Jenkins can't connect to Bitbucket at plugin initiation time, the Bitbucket config is broken until a one of the below mitigations is done. The isValid check is done here: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/blob/master/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java#L288

          Note that if the repository is invalid, there is no attempt to recreate it.

          We perform a weekly reboot of out Bitbucket and Jenkins servers at the same time, so we saw this error a lot because Bitbucket was down when the plugin was initializing.

          My request: Can there be some sort of progressive backoff implemented so that the job can self-heal?

          Here are the mitigations:

          • Stagger the reboots - this doesn't fix the issue permanently though because any downtime in Bitbucket could cause some pipelines to enter this state
          • Update the Bitbucket servers in the Jenkins global config. This appears to trigger a reload of all of the projects
          • (Fix a single job). Update the config for the job.

          With that said, I also don't know how can I programatically detect a job in this state.

          Scott added a comment - After weeks of toil, I think I finally pinpointed the problem! The problem is that if Jenkins can't connect to Bitbucket at plugin initiation time, the Bitbucket config is broken until a one of the below mitigations is done. The isValid check is done here: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/blob/master/src/main/java/com/atlassian/bitbucket/jenkins/internal/scm/BitbucketSCMSource.java#L288 Note that if the repository is invalid, there is no attempt to recreate it. We perform a weekly reboot of out Bitbucket and Jenkins servers at the same time, so we saw this error a lot because Bitbucket was down when the plugin was initializing. My request: Can there be some sort of progressive backoff implemented so that the job can self-heal? Here are the mitigations: Stagger the reboots - this doesn't fix the issue permanently though because any downtime in Bitbucket could cause some pipelines to enter this state Update the Bitbucket servers in the Jenkins global config. This appears to trigger a reload of all of the projects (Fix a single job). Update the config for the job. With that said, I also don't know how can I programatically detect a job in this state.

          Nick added a comment -

          This perfectly explains our situation.  Bitbucket and Jenkins on the same hardware.  After a reboot Jenkins will always complete launching before Bitbucker and therefore will be unavailable.

           

          Ironically, we hadn’t had problems for months, and then I needed to reboot the hardware earlier today and the problem resurfaced.  So your suggestion to back off and retry would be the best option.

          Nick added a comment - This perfectly explains our situation.  Bitbucket and Jenkins on the same hardware.  After a reboot Jenkins will always complete launching before Bitbucker and therefore will be unavailable.   Ironically, we hadn’t had problems for months, and then I needed to reboot the hardware earlier today and the problem resurfaced.  So your suggestion to back off and retry would be the best option.

          Scott added a comment -

          I submitted a PR for a potential fix, which my team has tested internally: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/pull/449.

          Scott added a comment - I submitted a PR for a potential fix, which my team has tested internally: https://github.com/jenkinsci/atlassian-bitbucket-server-integration-plugin/pull/449.

            Unassigned Unassigned
            passkit Nick
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: