Code changed in jenkins
User: R. Tyler Croy
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/b52d4f68547e2d09d50cc4127d37a153622b6c85
Log:
Ignore SCM API 2.0 related changes until the cause of JENKINS-41121

The basic idea of this is that the upgrades are sufficiently dangerous and *one
direction* (i.e. no rollback except for restoring from a pre-upgrade backup of
JENKINS_HOME) that it's best for Jenkins users to hide these versions until this
is figured out and made more safe.

See https://jenkins.io/blog/2017/01/17/scm-api-2/ for the list of affected
plugins.

Based on michaelneale/'s comment here, webhook triggering is broken for any branch with a "weird" character in it.

Code changed in jenkins
User: R. Tyler Croy
Path:
content/blog/2017/2017-01-17-scm-api-2.0-release.adoc
http://jenkins-ci.org/commit/jenkins.io/cceffabdd93994ec1d9f974e67fb927fb50ae496
Log:
Advise against upgrading plugins until JENKINS-41121 is resolved

This commit should be reverted when the fixes hit the update center

Code changed in jenkins
User: R. Tyler Croy
Path:
content/blog/2017/2017-01-17-scm-api-2.0-release.adoc
http://jenkins-ci.org/commit/jenkins.io/beac68e9fb301b1b3a6993ae066f44c16d5486a8
Log:
Merge pull request #550 from rtyler/41121

Advise against upgrading plugins until JENKINS-41121 is resolved

Compare: https://github.com/jenkins-infra/jenkins.io/compare/7dfbc443c845...beac68e9fb30

Stephen and I have been discussing this. Here are my notes from the discussion:

Impact

• The worst impact is limited to people using Org Folders with lots of open PR's
• All PR's on any project in an Org Folder must be rebuilt.
• This only affects users of Org Folders. Standalone multi-branch builds are unaffected.

Root Cause

The previous Branch API plugin didn't record sufficient information for us to migrate the state of the historical PR's. Stephen estimates that 5-10% of historical builds were built using incorrect data. We don't know if the last build reflects reality.

Possible fixes

• We can supress building the PR's but pretend you did if there's already a build of that PR.
• Alternatively, add a UX indicator to PR's which are in this indeterminate state instead of automatically launching the build.
  • This would probably also need UX actions for launching these builds to correct the state
• Alternatively, throttle the launching of the builds so that we progressively resolve the issue over time instead of all at once
• Some combination of the above that addresses build-storm issue which is endemic to org folders. (My personal favorite)

built using incorrect data

What incorrect data would that be? Can you give some concrete examples to help evaluate whether other fixes might be possible?

What incorrect data would that be? Can you give some concrete examples to help evaluate whether other fixes might be possible?

So there are a number of cases, the easiest to understand is the case of the merge commit PR.

You can edit the target branch of a PR on GitHub.

The old way that pull requests were tracked persisted the original target branch as part of the SCMHead but did not include that information in the equality contract for the SCMHead

Thus depending on the code path followed through Jenkins, you can either get a build result on the PR commit of success or failure that corresponds to a merge commit against the wrong target branch.

There are other cases, but they get more complex to follow and analyse

I triggered a rescan of a single multibranch project and it triggered a rebuild storm (not just org folder).

Mic's comments: an alternative experience is that things won't work and you have to trigger a rescan by hand. Result is a build storm either way.

Michael cancelled all the queued items and all the commit statuses are broken due to cancelled builds. The history of the PR is otherwise retained.

Mic says he saw this happen on a plain old multibranch project, not an org folder. As a branch source folder user, Mic had to do a reindex and this caused a build storm. If branch source is expected to be fine and that is not related to this issue, we may have seperate issue.

We need to clarify if this is addresses what Michael saw with his multi-branch project or if this is a separate issue.

Hmmm so that may be a separate issue with PRs only. Were any of the branches affected or was it just PRs

The current fix I have is https://github.com/jenkinsci/branch-api-plugin/pull/71 which will fix the build storm in org folders.

It is possible that there could be a partial build storm in multi-branch, that would be for PRs only if I'm interpreting the code paths correctly... not sure if I can dissect that out as we have a slightly different requirements to fix that

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/resources/artifact-ignores.properties
http://jenkins-ci.org/commit/backend-update-center2/80b552055b220f045c554a0de81f83159bebbdb5
Log:
Merge pull request #100 from jenkinsci/JENKINS-41121

Ignore SCM API 2.0 related changes until JENKINS-41121 is addressed

Compare: https://github.com/jenkinsci/backend-update-center2/compare/a6431a9d6790...80b552055b22

Current status: new metadata got generated, and waiting for mirrors to pick them up

To clarify KK's comments above, this release has been removed from update centers until we can address this issue and others.

FYI bwalding also saw this on a private instance with a large number of jobs and branches. All the branches retriggered automatically (he doesn't use PRs) with no code change. This could have been much worse as some branches represent deployments (and whilst it would deploy the same thing.. one does not simply redeploy for fun - and he has other gates in place that make it a non issue but it may be surprising behavior to many).

This was using a mutlibranch github project. Not org folder.

Using a multibranch pipeline project with the latest SCM API 2.0 release, we have also noticed PR's from contributors getting flagged as untrusted sources.. Despite the PR author having write OR admin privileges to the github project.

To test this.. Submit a PR with changes to a project's Jenkinsfile (add an echo or something).. Open up a PR and scan the repository.. In the scan log, your PR will look something like the following:

    Checking pull request #1817
    (not from a trusted source)
    Job name: PR-1817
      ‘Jenkinsfile’ found
    Met criteria

Since it's not a trusted source, when building this pull request, Jenkins will revert to using the Jenkinsfile on the base branch.. The log in the Jenkins PR job will look like this:

Loading trusted files from base branch dev at {commit} rather than {commit}

I can open up a defect for this bug, but it seems related to issues in scanning repository behavior.

Here is another critical break from this upgrade. Our CI is basically down with no recourse.
https://issues.jenkins-ci.org/browse/JENKINS-41207

patthiel, it would be quite useful if you could open another ticket for the trusted source problem. If it is actually a dupe, we can close it as such. But it really doesn't seem like the same bug now.

patthiel: https://issues.jenkins-ci.org/browse/JENKINS-40652

I am going to split this issue into two separate root causes.

The first root cause (easier to fix) affects only Organization Folders. The first time they are scanned - after upgrade - all the SCMSource implementations will be given new permanent IDs rather than getting a random ID with every scan. As an ID change triggers a rebuild, that is one cause of the build storm.

The second root cause is a side-effect of a bad decision in the early implementation of GitHub Branch Source and BitBucket Branch Source whereby rather than having a dedicated sub-class for the SCMHead, instead a implementation class from Git Plugin was reused. This would have the side-effect of causing SCMEvent triggers to incorrectly fire. In order to prevent that side-effect, the GitHub Branch Source has now migrated to a dedicated `BranchSCMHead` implementation class. The problem is that we cannot fix that through `readResolve` as the original class is still present and correct in the context of a Git Branch Source.

I am splitting off the first (simpler) issue and fixing that first. This issue will remain for the second (more complex) issue

https://issues.jenkins-ci.org/browse/JENKINS-41255 is the split issue

Code changed in jenkins
User: Stephen Connolly
Path:
docs/consumer.adoc
docs/implementation.adoc
src/main/java/jenkins/scm/api/SCMHeadMigration.java
http://jenkins-ci.org/commit/scm-api-plugin/adbf5db6847cbe38fe4aa25664d2f8bfec2eeef0
Log:
JENKINS-41121 Provide an API for SCMHead migration

Code changed in jenkins
User: Stephen Connolly
Path:
src/main/java/jenkins/scm/api/SCMHeadMigration.java
http://jenkins-ci.org/commit/scm-api-plugin/6f82bcb339d6980c6303b7b394bbf3c21d1d02ad
Log:
JENKINS-41121 Address review comments

Code changed in jenkins
User: Stephen Connolly
Path:
docs/consumer.adoc
docs/implementation.adoc
src/main/java/jenkins/scm/api/SCMHeadMigration.java
http://jenkins-ci.org/commit/scm-api-plugin/4b3eb6307d1e4df52c0f31a131f1f2f4fe283fe5
Log:
Merge pull request #24 from stephenc/jenkins-41121

JENKINS-41121 Provide an API for SCMHead migration

Compare: https://github.com/jenkinsci/scm-api-plugin/compare/f66f081cc3a3...4b3eb6307d1e

Code changed in jenkins
User: Stephen Connolly
Path:
pom.xml
src/main/java/jenkins/branch/MultiBranchProject.java
http://jenkins-ci.org/commit/branch-api-plugin/21cdf0dfe58dff855e68edcc733be988fef01fb2
Log:
JENKINS-41121 Consult the API for SCMHead migration on loading multibranch projects

Code changed in jenkins
User: Stephen Connolly
Path:
src/main/java/jenkins/branch/MultiBranchProject.java
http://jenkins-ci.org/commit/branch-api-plugin/bb15e39c80e5f6d5b4c3ee693ac1ed35062a7932
Log:
JENKINS-41121 Fix findbugs issue

Code changed in jenkins
User: Stephen Connolly
Path:
pom.xml
src/main/java/jenkins/branch/MultiBranchProject.java
http://jenkins-ci.org/commit/branch-api-plugin/c8d9b63809543e3d4c4aab59e7fa23d4175e8f87
Log:
Merge pull request #75 from stephenc/jenkins-41121

JENKINS-41121 Use the SCMHeadMigration API

Compare: https://github.com/jenkinsci/branch-api-plugin/compare/a06c22de5600...c8d9b6380954

Code changed in jenkins
User: Stephen Connolly
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/github_branch_source/BranchSCMHead.java
http://jenkins-ci.org/commit/github-branch-source-plugin/fd0d732116d2b940e4a8bc543e05e0c1c76b886e
Log:
JENKINS-41121 Implement the SCMHeadMigration API

Code changed in jenkins
User: Stephen Connolly
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/github_branch_source/BranchSCMHead.java
http://jenkins-ci.org/commit/github-branch-source-plugin/6c7ad6542eb6690a802f0d1a2b07944dc161ca29
Log:
Merge pull request #107 from stephenc/jenkins-41121

JENKINS-41121 Implement the SCMHeadMigration API

Compare: https://github.com/jenkinsci/github-branch-source-plugin/compare/4b195295fb03...6c7ad6542eb6

scm-api-2.0.2
git-2.6.4 or git-3.0.4
branch-api-2.0.2
bitbucket-branch-source-2.0.2
github-branch-source-2.0.1
cloudbees-folder-5.17
workflow-multibranch-2.11

morgan_goose The issue was resolved with releases in the experimental update center. The plugin release has been completed and all tests are successful, but is currently being held back from the UCs by https://github.com/jenkinsci/backend-update-center2/pull/109 until Monday to allow for better responsiveness in the event of further issues.

If you are eager to get the release, you can download this zip https://www.dropbox.com/s/5g4qm8gms53pxqc/scm-api.zip?dl=0 which has the latest versions of all the complete transitive dependency graph of Blue Ocean and SCM API (as Blue Ocean is required to be updated if you update SCM API)

Otherwise, just wait until Monday

Here is the draft of the blog post to accompany Monday's release into the UC https://github.com/jenkins-infra/jenkins.io/pull/580

But there are already plugins available for update that explicitly require this version of the scm plugin. Withholding the release is breaking Jenkins stacks. Letting users only upgrade partially, causes Jenkins to suggests resolution of "old data" via Deleting said data.

If a core plugin is the hub of any number of spoke plugins that require it's use, it's seem pretty disruptive to not release them all at the same time? I've already grabbed the two hpi files, and installed manually:

• github-organization-folder-1.6.hpi
• scm-api-2.0.2.hpi

from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/

Was this release standard? Or did it follow a different track than usual?

morgan_goose

Manually downloading plugins from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ is neither the normal way to install plugins nor the recommended way.

When you manually download plugins you have to take responsibility to manually download the required dependencies.

For the SCM API and Blue Ocean related updates that is potentially a list of updates that could grow to 72 plugins depending on what plugins and versions you have installed prior to upgrading... though an otherwise up to date system might only need 24 plugins... and if you do not have Blue Ocean installed that number could drop to somewhere between 7 and 9.

Given that you have manually installed more than one of the plugins, my recommendation is that you download the ZIP file that I gave a link to in https://issues.jenkins-ci.org/browse/JENKINS-41661 and use the plugins in that ZIP file to update your instance.

For future reference, the recommended path for installing plugins is via the update center. There are safety checks that the update center applies to help prevent issues. While we can agree that there are more additional safety checks needed, the limited checks it does have are better than no checks which is the net effect of manual downloading.

I can tell you that I did not download any plugins by hand, and only used the update center. And still wound up reading this issue.

Echo everything that Morgan Goose said. I downgraded some plugins because that appeared to be the recommended path, and I only narrowly avoided deleting my settings after reading the same warnings that you saw about old data. Maybe I caught the upgrade moments before you blacklisted the 2.0.1 update? Not sure. The timing is about right though.

Thanks for reposting the ZIP link. I see that you're planning a Monday release, I will wait for that, thanks for providing a clear(ish) path in the issue documentation. FWIW none of the plugins that complained at me about bad dependency versions, actually appeared to be malfunctioning, but I am not using Blue Ocean or advanced GitHub API plugins yet.

kingdonb you have a different case from Morgan.

Morgan said:

I've already grabbed the two hpi files, and installed manually:

• github-organization-folder-1.6.hpi
• scm-api-2.0.2.hpi
  from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/

Whereas you downloaded the plugins via the recommended path (and were just bit by the issues described in JENKINS-41661 which is where I am complaining that the UC does not do enough validation. The UC method does at least try to upgrade all the required dependencies... it should grey out plugins that cannot be installed IMHO)

Kingdon, I recommend reading https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-41661+Post-Mortem for more details on the JENKINS-41661 issue that affected you.

The plugins will all be released on Monday.

Sorry for any inconvenience caused.

Yeah, sounds like you're on top of things. Lesson learned: full backups immediately before every upgrade!

No trouble at all, thanks for your contribution!

kingdonb

This was a particularly troublesome set of upgrades. Due to the mistakes made in GHBS and BBBS (because of lack of documentation) we had to break data compatibility in a particularly complicated way that prevented the normal evolution mode in Jenkins.

Hopefully we will not be doing that again any time in the near future

stephenconnolly I'd like to clarify, that I only installed those two plugins manually after multiple plugins failed to load saying that these versions were required. All of those complaining plugins were shown as having updates available and were updated in the suggested way (via the plugin update center web UI).