Could you please paste the JSON response from the REST API call to .../runs/$runId?

This looks like it's an underlying behavior in Pipeline.

I think this is a serious scaling problem that a tiny shared library change can trigger a huge number of jobs. Any chance an improvement to make rebuilds optional gets picked up? I know that in our deployment we won't be able to afford such rebuilds.

Agreed ... we're looking to shared libraries to augment existing Jenkinsfile directives/steps (such as cleaner artifactory integration or change management activities in JIRA not possible with the present JIRA plugin) 

We really don't want to have commits to the implicitly loaded shared library (or even explicitly where a project does that, but implicit is the real problem as it affects all jobs) to add a new feature, or a minor bugfix, to display in the changelog of jobs much less trigger jobs across the estate.

We're using the Bitbucket source plugin with webhook plugin on Bitbucket and a very basic global shared library modern SCM (git) configuration for this.

I'll be doing some digging for any possible workarounds etc today.

Actually ... on now doing some initial investigation

I don't think triggering is actually happening, just the changes of the shared library being included in the changelog

tsniatowski are you sure you saw actual triggering of jobs rather than just a  job properly triggered then also including the changelog of the library in the job detail?

Now I'm certain the webhook from Bitbucket is not triggering the job when a change is made to the shared library, but not the actual job repo.

It's also not triggering on a general reindexing by the Bitbucket plugin of that project

It is triggering a build if the Jenkinsfile has properties set to carry out an SCM poll, but this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression.

This does still have library changes appear in the recent changes of the job though.

It is triggering a build if the Jenkinsfile has properties set to carry out an SCM poll, but this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression.

I've seen similar suggestions for workarounds in other issues (JENKINS-39615), but some official answer would be nice, especially as many of us use SCM polling (for one reason or another) and the global pipeline library defaults to settings that will trigger builds which use polling.

jglick could you please take a look at my PR for the changeset part of this.

https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/34

Testing this in our internal test instance it behaves as expected so far as I can tell with commits to the shared library not included in the recent changes of teh job or the change details of a build when the checkbox is unticked.

With the checkbox ticked it behaves as it currently does with the changeset of the shared library, if it has changed so the changeset exists, being added to the changes page of the build and appearing in recent changes of the job.

I've defaulted it to show include the changeset of the library as that matches the current behaviour.

hogarthj on your test instance, if your push some modifications on a shared library, does it trigger a job using this library with SCM polling activated (and your checkbox unticked) ?

mathieulaude when digging in to this I realised that particular problem isn't actually an issue with this plugin.

You can see here that the plugin actually sets setPoll to false.

It's actually SCMTrigger that is responsible for that, but I don't (from a cursory look at the code) think it has any way of differentiating an SCM associated with a Job from a library versus any other scm checkout.

For now I suggest using the workaround on that with the "skip if path affected" or "skip based on message" set to a wildcard of everything.

I'm hoping this is acceptable to Jesse etc at least though so the confusion from mixed changesets can at least be addressed as we'd like to see this change in our workplace.

OK, thanks for the explanation. I'll keep the workaround for now.

For us it is not only the triggers that cause the pain - we use the changes as reported also to set the build name (and do some other magic with the derived JIRA IDs). Obviously those changes can be filtered, but it results in unnecessary complicated code and inconsistent UI (since the changes as reported always show up the changes list in the very left column of the pipeline stage view).

If someone could merge https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/34 it would be highly appreciated!

I can definitely confirm this problem is real and incredibly challenging to deal with (running an instance with thousands of jobs, of which hundreds use the library to do things like deploys etc ).  

I created a simple test case where I make a Pipeline job and just have it use a simple pipeline:

echo "Hello World"!

I then configure my global library and implicitly included.  On the piplein job I turn on polling. Then I commit a simple readme chagne to my pipeline library and watch polling trip my job when it changes.

I even have :

"Ignore on push notifications" AND

"Don't trigger a build on commit notifications" (the Advanced clone behaviors) set

Changes to the library will /always/ trigger the job.  Which makes the feature basically useless at scale. So I'm all for the PR above that allows us to disable including it in the polling logs from the central config page in the implicitly loaded case.  Without it, we basically have to sit and hope that some day this will work correctly. We're falling back to the old way of having the library pushed to the central location via the SSH/exposed git endpoint (which never had this problem because it was never included in the polling logs).

maxfields2000 - did you try the workaround I suggested a little way above?

In the testing I did this way did disable polling of the share library when polling was set on the job itself.

For now I suggest using the workaround on that with the "skip if path affected" or "skip based on message" set to a wildcard of everything.

That workaround does not work if you have no workspace. The git plugin requires creating a workspace in order to do that kind of filtering.  We have a use case where in many cases there is no workspace (it's by design, we dynamically create space for jobs to run) until the job is triggered.  When you use the filtering feature and you have no worksapce it will trigger your job even if your central scripts don't change (because it has to make a workspace) which then results in infinitely triggering jobs. So really this is a direct ask to please make a simple toggle that says "Do not include in polling or change history"  or better yet have that the default behavior and have an option that says "include in polling/change history".

Prior to this "Global Library" feature, the "central library" approach by using SSH/Git server did not include itself in any polling logs or results. This new implementation of Global Libraries should also not include itself, or it's changelogs, as part of the "job" by default (I can see why you might want to have it do so in some cases)

Otherwise, this feature is a nice toy but not really scalable (you really don't want 5,000 jobs firing because you changed a shared function).  Filters (even if you have permanent workspaces) are wholey inconvenient. The work arounds here for others in this shape are to revert to using the old way of globally sharing code (which isn't branch aware or referenceable). Be advised though that any functions in that library that have the same name as other libraries will trump the other libraries.

For now we're furiously working to tell our job owners to convert to web hooks entirely and stop using Polling (I wish I could disable polling entirely as a jenkins wide thing).

this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression

The above workaround doesn't seem to be working in Jenkins 2.58. Is there a specific version jenkins and a plugin needed for this workaround?

Workaround suggested doesn't work. Using Jenkins 2.60.1. This is a critical issue for us as we are not able to scale using shared library. Hope this issue get's fixed soon.

All workarounds suggested only make the issue worse for us. With no nothing additional configured in settings for the global shared library, Jenkins will kick every build using that shared library when any change is committed. When any kind of filtering is attempted (Polling ignores commits, Don't trigger a build on commits, etc), Jenkins will kick every build using that shared library every single polling interval, regardless of whether or not anything changed.

This is a critical issue for; we have completely stopped using shared libraries because of this issue.

We found a workaround that works for us: stop controlling the shared library as a GitHub project and control it in a legacy SCM (i.e. Perforce) instead. Polling directives (Polling ignores changes based on <X>) do seem to work just fine when we house the shared library in a legacy SCM, whereas those directives don't seem to work with the GitHub retrieval implementation (or make it worse, as described above).

Wondering if "minor" is the correct priority / criticality for this.

After all, this renders the whole global-shared-library feature useless. It's the showstopper why we didn't upgrade our jenkins since 2.32.1...

Does anyone at cloudbees actually use global shared libraries like this, or are we doing it wrong?

/ping jglick

tknerr The best way to get rid of this issue is changing the pipeline trigger model to webhooks, this way there is no trigger on the Library repo itself, just in the main projects. But as this is an issue for the polling method. It does not apply... but it's fine to remind it I guess.

Although the issue of not been able to refine changeset still applies when using webhooks.

I wasn't aware this is a polling only issue, thanks for the pointer xoue!

Still, we can't use webhooks due to our firewall setup currently  

I am aware of the issue but have not had a chance to work on it.

From my understanding (with the git/github plugins anyways), a PUSH event from the webhook triggers the polling mechanism. This can be seen in the polling log on a job that is only configured for GHPush. The webhook was received:

Jul 27, 2017 11:26:31 AM org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
INFO: Received PUSH for https://ghe.domain.com/my-org/code-repo from 10.52.7.244 ? http://jenkins.domain.com:8080/build-system-1/github-webhook/
Jul 27, 2017 11:26:31 AM com.cloudbees.jenkins.GitHubPushTrigger$1 run
INFO: SCM changes detected in Pipelines/code-repo. Triggering #5

Then this output can be found in the polling log:

Polling Log
View as plain text
This page captures the polling log that triggered this build.
Started on Jul 27, 2017 11:26:24 AM
Started by event from 10.52.7.244 ? http://jenkins.domain.com:8080/build-system-1/github-webhook/ on Thu Jul 27 11:26:24 MDT 2017
Using strategy: Specific revision
[poll] Last Built Revision: Revision caabb3d7fd98ea63ad76339eed6633590e78072b (master)
using GIT_SSH to set credentials GitHub SSH Key ssh-poller-creds
 > git ls-remote -h git@ghe.domain.com:my-org/jenkins-libraries.git # timeout=10
Found 1 remote heads on git@ghe.domain.com:my-org/jenkins-libraries.git
[poll] Latest remote head revision on refs/heads/master is: 7ae6d912bcad95b6baffcd9bd24580b3dd1cf78d
Using strategy: Default
[poll] Last Built Revision: Revision 6e872eb81537e239fc7b00c8a1ac5878aafabff2 (refs/remotes/origin/master)
using GIT_SSH to set credentials GitHub SSH Key ssh-poller-creds
 > git ls-remote -h git@ghe.domain.com:cban/code-repo.git # timeout=10
Found 5 remote heads on git@ghe.domain.com:cban/code-repo.git
[poll] Latest remote head revision on refs/heads/master is: 6e872eb81537e239fc7b00c8a1ac5878aafabff2 - already built by 4
Done. Took 0.24 sec
Changes found

We may have a misconfiguration, but the webhook from a PR (from GHPRB) also triggers this mechanism so on every PR build, the master kicks off too when there are library changes. 

abayer to investigate (may punt to sam if needed)

There is a proposed PR; last time I had any time to look at it, it needed some design work. Not sure of current status.

With the latest update of Git Plugin, the trick of setting the "polling ignores commits with certain messages" option (as described by hogarthj in comment above) is not possible anymore.

Every push on my shared library triggers my multi-branch pipeline again this morning (15 branches of 1 hour each)... This issue becomes critical for me.

It is critical issue for my setup too.

Problem is even more complicated when custom branch pipelines are using shared library with checkout command for few git projects in parallel section (some of them are calling checkout with poll:true and some with poll:false). I use checkout command with refspec + checkout to separate directotry + repo name because of repeated checkouts. In such scenario polling log is pretty hard to use - it shows only information "Changes found" and a lot of entries "Ignoring refs/heads/...as it doesn't match any of the configured refspecs" - it is confusing as today single pipeline was triggered 30 times in two hours when number of commits was only few. How do you advise to narrow down the issue or potential not optimal using of checkout commands.

I have same problem.

At a minimum it would be nice to changelog and polling toggles for shared library. Only having ignore push notification isn't useful for polling.

Merged the PR, releasing shortly.

Released as workflow-cps-global-lib 2.9.

Upgraded to 2.9, but still experience the same problem. See attached screenshot of the configuration of the global library and of one of the dozens of triggered builds.


I can confirm I do not see the library commit in the triggered builds changelogs, but that seemed a minor problem with respect to this ticket's description, which is not to trigger all jobs everywhere.

I tried switching Modern SCM from Git to Github and back, and recreating the configuration of the global library fearing some state was lingering from the previous version of the plugin.

Sample output from one of the triggered builds demonstrating the commit message is coming from the shared library: https://gist.github.com/giorgiosironi/ef4f326d0e959cbd826d9a6cef769b44

Rolled back to the internal /workflowLibs.git repository approach, the only workaround I know.

Doesn't solve the problem - for some reason when I push to the Github equivalent of the internal /workflowLibs.git (used as a backup: https://github.com/elifesciences/elife-jenkins-workflow-libs), a web hook triggers all builds again. Now going through organizational and repository hooks to check what's happening.

The log says:

Sep 25, 2017 9:55:48 AM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent

Received PUSH for https://github.com/elifesciences/elife-jenkins-workflow-libs from 192.30.252.40 ⇒ http://ci--alfred.elifesciences.org:8080/github-webhook/

Sep 25, 2017 9:55:48 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

SCM changes detected in library-api-validator-php. Triggering #22

Sep 25, 2017 9:55:49 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

SCM changes detected in library-patterns-php. Triggering #434

Sep 25, 2017 9:55:49 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

Since the web hook is organization-wide, I don't even know how does it understand that the repository is a global workflow library. I have gone through workspaces to delete the ...@libs folders to no avail.

Deleted all workspaces too, no change. The log of a triggered build continues to say:

 > git ls-remote -h git@github.com:elifesciences/elife-jenkins-workflow-libs.git # timeout=10
Found 1 remote heads on git@github.com:elifesciences/elife-jenkins-workflow-libs.git
[poll] Latest remote head revision on refs/heads/master is: 9f5110ead60321557cdc0e5394ec299d70983938
Using strategy: Default
[poll] Last Built Revision: Revision 966f0ebb926fbe6e2db5d2de68a86aec4ff86676 (refs/remotes/origin/develop)

and I don't know where it's getting that repository URL from.

I have found these references:

elife@ci--alfred:/var/lib/jenkins$ grep -r elife-jenkins-workflow-libs jobs/test-journal/builds/1056/build.xml                            
        <string>git@github.com:elifesciences/elife-jenkins-workflow-libs.git</string>
            <url>git@github.com:elifesciences/elife-jenkins-workflow-libs.git</url>

which are on a previous build rather than on the jobs definition, it is possible they are used to determine whether to trigger the job again. Will try to remove it on a test job.

To delete these references, I found out you have to run at least 1 (successful?) build for each job without the global pipeline library being defined in Global Settings. This seems to remove from the jobs/ metadata the reference to the external repository, so that when the Github hook triggers it is ignored. I am now running all builds once to clear all references before I am able to make changes to the library and push it to /workflowLibs.git.

Confirmed that poll SCM jobs are still triggered by every change to a shared library. What's the way forward on this? There are several duplicate JIRAs which describe this issue, but I don't see any open, active ones. Does this one need to be reopened, or does a new Major/Critical Bug need to be written?

dolphy possibly a new one (with repro details) cc abayer

As far as I can tell, the fix works. I noticed that existing jobs will need to be built at least once for this to take effect. This will "fix" the metadata recorded the SCM used. So when you upgrade to that version, the first next change to the shared library code might still triggers many jobs. Next changes will not.

ah thanks allan_burdajewicz!

cc dolphy

allan_burdajewicz michaelneale

You're right, and I'm wrong! I had based my comment solely off of seeing the jobs kick again, but never verified that they will continually kick. After the first time, they no longer trigger on a shared library commit.

Sorry for the hasty assumption, and thanks!

The global configuration option only for works for implicitly loaded libraries. If I include a library with the library step, I have to specify

library identifier: "MyLibrary", changelog: false

I would expect the default behavior

library "MyLibrary"

to be controlled by the global LibraryConfiguration option. As it is, I have to edit a large number of Jenkinsfiles to enable the new feature.

It seems like my expected behavior might have been intended originally. In the implementation in LibraryStep.java

public static class Execution extends AbstractSynchronousNonBlockingStepExecution<LoadedClasses> {
    @Override protected LoadedClasses run() throws Exception {
        // ...
        Boolean changelog = step.getChangelog(); // returns default value LibraryStep.changelog = true, if not provided
        // ...
        changelog = cfg.defaultedChangelogs(changelog); // returns changelog if != null, else global configuration option
        // ...
    }
}

a function is called that loads the global configuration value if the argument is null, but due to the default initialization, this statement only has effect if changelog = null is specified explicitly. However, a comment for LibraryStep.setChangelog states: default to including changes of the library in job recent changes.

Is this working as intended or should I open a PR for this?

mnsa do you mind opening that as a new feature JIRA for this and a PR that goes with it (if you want to) - and could be discussed there (or at least the ticket, where can be discussed). Changing default usually needs discussion but it seems reasonable in this case that the default is probably not right. 

With workflow-cps-global-lib 2.9 still seeing that jobs are re-built multiple times, with the option unchecked to include library changes in the config UI. I would agree that changing the default makes sense in this case because it is very unintuitive and can cause pretty large issues to your cluster throughput and stability, while if you wanted it and it gets turned off, you can easily turn it on.

From JENKINS-41497, "The global configuration option only for works for implicitly loaded libraries." For non-implicit libraries, is it expected that we have to set
changelog: false
..in every Jenkinsfile? If so, then the global setting should be a child checkbox under the implicit checkbox.

I opened issue JENKINS-48487 for this specific problem. It is actually only a problem of the the library step, using the '@Library' decorator, it works fine. For the library step you would have to specify changelog: false in every Jenkinsfile (and change it to true, if you should change your mind later on). Since it works for the decorator syntax, I don't think it should be a child checkbox of the implicit checkbox. I filed it as a bug because the behavior is different between the decorator and the step.

I see this issue once again with Jenkins 2.107.2, on 2.107.1 it is working just fine

This bug is still present on version 2.116. I found that the workaround mentioned here is the way to go for the moment. Thanks allan_burdajewicz!