Search

Out of the people who are experiencing this, are you using jobDSL to create and update your multibranch pipelines?

Yes, I am using JobDSL to create and update those jobs.

I found out the route cause of the problem for my case.

We are provisioning our Jenkins instance with Ansible and we are generating the jobs with the REST API passing the config.xml in the body (as documented in the Jenkins REST API documentation). The problem was that we passed a wrong scm id in the branch source information :

<jenkins.branch.BranchSource>
  <source class="jenkins.plugins.git.GitSCMSource" plugin="git@3.5.1">
    <id>xxx</id>
    <remote>xxx</remote>
    <credentialsId>xxx</credentialsId>
    <traits>
      <jenkins.plugins.git.traits.BranchDiscoveryTrait/>
    </traits>
  </source>
  <strategy class="jenkins.branch.DefaultBranchPropertyStrategy">
    <properties class="empty-list"/>
  </strategy>
</jenkins.branch.BranchSource>

What solved it for us was to simply remove the <id> part of the git scm source and Jenkins generated the right one.

I looked into it, and I think we're hitting JENKINS-43693 in some form. I believe jobDSL is is possibly re-generating the ID of the SCMSource every run, and after that the branch indexes are lost, which is why Jenkins is giving us this error.

chancez I think actually this is related to JENKINS-39682 because it appears that when you trigger a full re-index(like you would do after a save of the configuration for a multibranch job) the issue goes away. So I think that just fixing the DSL code to call an index after configuring the job itself should fix this issue.

Would that work?

We are running into this issue as well, and although we could simply re-index our multibranch project, it's really annoying because it triggers a full rebuild of every branch of the project.  The real problem is https://issues.jenkins-ci.org/browse/JENKINS-43693 which I don't think solved the issue of the <id> field for each branch of a multibranch project getting updated when a dsl job gets kicked off.

Just as a note, there are recent comments in the ticket above from folks still noticing the issue after the 'fix' was put in place.

ataylor Having to re-index immediately after is a stop-gap, as binelson mentioned, in some cases it causes unnecessary re-builds, depending on the job behavior (a multibranch job set to merge to HEAD means all PRs get re-triggered, which isn't always what we want).

Additionally, setting the `<id>` field explicitly isn't making a difference in my testing.

In my case, adding an ID to JobDSL script actually solved couple of weird issues (one multibranch job was automatically triggered after branch indexing, another one had above error).

I'm seeing this, but I found a repeatable trigger.

I can provide more details but in short it goes like this:

• Build out a pipeline in blue ocean. It'll work fine, consistently.
• Add the Hashicorp Vault Plugin.
• All your stuff breaks with complaints about "exact tip revision"
• Go to the repo in the management console, hit "scan repository now"
• Things work again

Now, I don't specifically fault the vault plugin with this. Especially since I'm not at the point where I'm directly using it. It makes me think the plugin installation process is breaking repo metadata or something and jenkins isn't cleanly recovering? I'm unclear on the precise mechanics and don't want to burn too much time on it since I found a (for me) painless workaround for something that (for me) has a very specific and rare trigger.

I'm seeing the same issues as jowr every time I use the build ocean pipeline builder and then attempt to install a plugin. In effect: Blue Ocean Pipeline Builder renders my builds inoperable.

In my case, adding an ID to JobDSL script actually solved couple of weird issues (one multibranch job was automatically triggered after branch indexing, another one had above error).

Every branch source is required to have an non-null ID. The ID must be unique across all the branch sources within the multibranch job. In other words if your job has one and only one source, then just give it the id of i-am-unique. If will not matter if you have 50 or 5000 multibranch jobs each with single different source and all those sources with an id of i-am-unique as the ID is used to differentiate the origin of branches within the single multibranch project, so we do not care if other different multibranch projects have sources with the same id. https://issues.jenkins-ci.org/browse/JENKINS-48571 contains comments from me that may help

unlikelyzero you are in the first cohort I describe from my analysis in https://issues.jenkins-ci.org/browse/JENKINS-48571?focusedCommentId=329111&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-329111 namely you are being caught by the bug in BlueOcean where it is not setting the ID.

To be clear, not assigning an ID is a bug in BlueOcean as SCM API has no way to determine if a SCMHead of master is coming from one source or another source in the event that you have two sources configured... previously BlueOcean was relying on a side-effect that would accidentally assign a non-null id before the configuration was saved... however, what this means is that every time BlueOcean updated the job, a new randomly generated id would be assigned, and consequently a rebuild storm would be triggered on the next full scan. Because reconfiguring a job should not result in a rebuild storm (as distinct from a build storm), and the rebuild storm is a result of the failure to supply a non-null id that reflects the fact that in a BlueOcean managed multibranch project, there is one and only one source then the failure to supply an id is a bug in BlueOcean.

In otherwords, the accidental side-effect that BlueOcean was relying on was causing a more subtle bug that would have been hard for users to identify...

• a build storm occurs when you change the configuration to discover new branches that were not discovered before
• a rebuild storm occurs when you change the configuration and all the existing branches get rebuilt

There are potentially some cases where you might actually want a rebuild storm (e.g. say you added a "clean before checkout" trait) but the branch-api plugin will err on the side of caution and only rebuild if the ID has changed, assuming that changes impacting the SCMs generated by a SCMSource with the same ID will be caught by users retriggering the branches that need a retrigger... after all branch-api does not know which changes within an SCMSource are significant... but if you Add a new SCMSource and remove the old one (as distinct from modifying the existing one's configuration)... well that is a significant change... looks like it could be a whole different repository, we need to rebuild that.

The behaviour that BlueOcean is manifesting - from the point of view of branch-api - looks indistinguishable from Remove-followed-by-Add as distinct from Update... consequently branch-api will interpret that as "The source has changed, rebuild everything". BlueOcean should probably just be specifying an ID of blueocean and that would fix the issues for BlueOcean users.

The good news for BlueOcean users is the workaround (for the tip revision error) is simple, namely save the job in the classic UI after creation by BlueOcean... when you load the classic config screen, that will round-trip the randomly assigned ID that is stored in memory but has missed being saved to disk and persist it to disk. Sadly, if you subsequently modify the job using BlueOcean, you will again need to save the job in the classic UI again, but you will still have the side-effect of a build storm as BlueOcean's update will have assigned an id of null and the first call to SCMSource.getId() will have generated a new random id which is highly unlikely to match the previous random id and consequently a rebuild storm will occur.