Could you please paste the JSON response from the REST API call to .../runs/$runId?

This looks like it's an underlying behavior in Pipeline.

As designed, but an RFE to disable changelog and/or polling for library checkouts.

I think this is a serious scaling problem that a tiny shared library change can trigger a huge number of jobs. Any chance an improvement to make rebuilds optional gets picked up? I know that in our deployment we won't be able to afford such rebuilds.

Agreed ... we're looking to shared libraries to augment existing Jenkinsfile directives/steps (such as cleaner artifactory integration or change management activities in JIRA not possible with the present JIRA plugin) 

We really don't want to have commits to the implicitly loaded shared library (or even explicitly where a project does that, but implicit is the real problem as it affects all jobs) to add a new feature, or a minor bugfix, to display in the changelog of jobs much less trigger jobs across the estate.

We're using the Bitbucket source plugin with webhook plugin on Bitbucket and a very basic global shared library modern SCM (git) configuration for this.

I'll be doing some digging for any possible workarounds etc today.

Actually ... on now doing some initial investigation

I don't think triggering is actually happening, just the changes of the shared library being included in the changelog

tsniatowski are you sure you saw actual triggering of jobs rather than just a  job properly triggered then also including the changelog of the library in the job detail?

Now I'm certain the webhook from Bitbucket is not triggering the job when a change is made to the shared library, but not the actual job repo.

It's also not triggering on a general reindexing by the Bitbucket plugin of that project

It is triggering a build if the Jenkinsfile has properties set to carry out an SCM poll, but this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression.

This does still have library changes appear in the recent changes of the job though.

It is triggering a build if the Jenkinsfile has properties set to carry out an SCM poll, but this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression.

I've seen similar suggestions for workarounds in other issues (JENKINS-39615), but some official answer would be nice, especially as many of us use SCM polling (for one reason or another) and the global pipeline library defaults to settings that will trigger builds which use polling.

jglick could you please take a look at my PR for the changeset part of this.

https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/34

Testing this in our internal test instance it behaves as expected so far as I can tell with commits to the shared library not included in the recent changes of teh job or the change details of a build when the checkbox is unticked.

With the checkbox ticked it behaves as it currently does with the changeset of the shared library, if it has changed so the changeset exists, being added to the changes page of the build and appearing in recent changes of the job.

I've defaulted it to show include the changeset of the library as that matches the current behaviour.

hogarthj on your test instance, if your push some modifications on a shared library, does it trigger a job using this library with SCM polling activated (and your checkbox unticked) ?

mathieulaude when digging in to this I realised that particular problem isn't actually an issue with this plugin.

You can see here that the plugin actually sets setPoll to false.

It's actually SCMTrigger that is responsible for that, but I don't (from a cursory look at the code) think it has any way of differentiating an SCM associated with a Job from a library versus any other scm checkout.

For now I suggest using the workaround on that with the "skip if path affected" or "skip based on message" set to a wildcard of everything.

I'm hoping this is acceptable to Jesse etc at least though so the confusion from mixed changesets can at least be addressed as we'd like to see this change in our workplace.

OK, thanks for the explanation. I'll keep the workaround for now.

For us it is not only the triggers that cause the pain - we use the changes as reported also to set the build name (and do some other magic with the derived JIRA IDs). Obviously those changes can be filtered, but it results in unnecessary complicated code and inconsistent UI (since the changes as reported always show up the changes list in the very left column of the pipeline stage view).

If someone could merge https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/34 it would be highly appreciated!

I can definitely confirm this problem is real and incredibly challenging to deal with (running an instance with thousands of jobs, of which hundreds use the library to do things like deploys etc ).  

I created a simple test case where I make a Pipeline job and just have it use a simple pipeline:

echo "Hello World"!

I then configure my global library and implicitly included.  On the piplein job I turn on polling. Then I commit a simple readme chagne to my pipeline library and watch polling trip my job when it changes.

I even have :

"Ignore on push notifications" AND

"Don't trigger a build on commit notifications" (the Advanced clone behaviors) set

Changes to the library will /always/ trigger the job.  Which makes the feature basically useless at scale. So I'm all for the PR above that allows us to disable including it in the polling logs from the central config page in the implicitly loaded case.  Without it, we basically have to sit and hope that some day this will work correctly. We're falling back to the old way of having the library pushed to the central location via the SSH/exposed git endpoint (which never had this problem because it was never included in the polling logs).

maxfields2000 - did you try the workaround I suggested a little way above?

In the testing I did this way did disable polling of the share library when polling was set on the job itself.

For now I suggest using the workaround on that with the "skip if path affected" or "skip based on message" set to a wildcard of everything.

That workaround does not work if you have no workspace. The git plugin requires creating a workspace in order to do that kind of filtering.  We have a use case where in many cases there is no workspace (it's by design, we dynamically create space for jobs to run) until the job is triggered.  When you use the filtering feature and you have no worksapce it will trigger your job even if your central scripts don't change (because it has to make a workspace) which then results in infinitely triggering jobs. So really this is a direct ask to please make a simple toggle that says "Do not include in polling or change history"  or better yet have that the default behavior and have an option that says "include in polling/change history".

Prior to this "Global Library" feature, the "central library" approach by using SSH/Git server did not include itself in any polling logs or results. This new implementation of Global Libraries should also not include itself, or it's changelogs, as part of the "job" by default (I can see why you might want to have it do so in some cases)

Otherwise, this feature is a nice toy but not really scalable (you really don't want 5,000 jobs firing because you changed a shared function).  Filters (even if you have permanent workspaces) are wholey inconvenient. The work arounds here for others in this shape are to revert to using the old way of globally sharing code (which isn't branch aware or referenceable). Be advised though that any functions in that library that have the same name as other libraries will trump the other libraries.

For now we're furiously working to tell our job owners to convert to web hooks entirely and stop using Polling (I wish I could disable polling entirely as a jenkins wide thing).

this can be mitigated in configuring the global library but configuring the additional behaviour "polling ignores commits with certain messages" on the SCM config for the global library with a '.*' expression

The above workaround doesn't seem to be working in Jenkins 2.58. Is there a specific version jenkins and a plugin needed for this workaround?

Workaround suggested doesn't work. Using Jenkins 2.60.1. This is a critical issue for us as we are not able to scale using shared library. Hope this issue get's fixed soon.

All workarounds suggested only make the issue worse for us. With no nothing additional configured in settings for the global shared library, Jenkins will kick every build using that shared library when any change is committed. When any kind of filtering is attempted (Polling ignores commits, Don't trigger a build on commits, etc), Jenkins will kick every build using that shared library every single polling interval, regardless of whether or not anything changed.

This is a critical issue for; we have completely stopped using shared libraries because of this issue.

We found a workaround that works for us: stop controlling the shared library as a GitHub project and control it in a legacy SCM (i.e. Perforce) instead. Polling directives (Polling ignores changes based on <X>) do seem to work just fine when we house the shared library in a legacy SCM, whereas those directives don't seem to work with the GitHub retrieval implementation (or make it worse, as described above).

Wondering if "minor" is the correct priority / criticality for this.

After all, this renders the whole global-shared-library feature useless. It's the showstopper why we didn't upgrade our jenkins since 2.32.1...

Does anyone at cloudbees actually use global shared libraries like this, or are we doing it wrong?

/ping jglick

tknerr The best way to get rid of this issue is changing the pipeline trigger model to webhooks, this way there is no trigger on the Library repo itself, just in the main projects. But as this is an issue for the polling method. It does not apply... but it's fine to remind it I guess.

Although the issue of not been able to refine changeset still applies when using webhooks.

I wasn't aware this is a polling only issue, thanks for the pointer xoue!

Still, we can't use webhooks due to our firewall setup currently  

I am aware of the issue but have not had a chance to work on it.

abayer to investigate (may punt to sam if needed)

There is a proposed PR; last time I had any time to look at it, it needed some design work. Not sure of current status.

With the latest update of Git Plugin, the trick of setting the "polling ignores commits with certain messages" option (as described by hogarthj in comment above) is not possible anymore.

Every push on my shared library triggers my multi-branch pipeline again this morning (15 branches of 1 hour each)... This issue becomes critical for me.

It is critical issue for my setup too.

Problem is even more complicated when custom branch pipelines are using shared library with checkout command for few git projects in parallel section (some of them are calling checkout with poll:true and some with poll:false). I use checkout command with refspec + checkout to separate directotry + repo name because of repeated checkouts. In such scenario polling log is pretty hard to use - it shows only information "Changes found" and a lot of entries "Ignoring refs/heads/...as it doesn't match any of the configured refspecs" - it is confusing as today single pipeline was triggered 30 times in two hours when number of commits was only few. How do you advise to narrow down the issue or potential not optimal using of checkout commands.

I have same problem.

At a minimum it would be nice to changelog and polling toggles for shared library. Only having ignore push notification isn't useful for polling.

Merged the PR, releasing shortly.

Released as workflow-cps-global-lib 2.9.

Upgraded to 2.9, but still experience the same problem. See attached screenshot of the configuration of the global library and of one of the dozens of triggered builds.


I can confirm I do not see the library commit in the triggered builds changelogs, but that seemed a minor problem with respect to this ticket's description, which is not to trigger all jobs everywhere.

I tried switching Modern SCM from Git to Github and back, and recreating the configuration of the global library fearing some state was lingering from the previous version of the plugin.

Sample output from one of the triggered builds demonstrating the commit message is coming from the shared library: https://gist.github.com/giorgiosironi/ef4f326d0e959cbd826d9a6cef769b44

Rolled back to the internal /workflowLibs.git repository approach, the only workaround I know.

Doesn't solve the problem - for some reason when I push to the Github equivalent of the internal /workflowLibs.git (used as a backup: https://github.com/elifesciences/elife-jenkins-workflow-libs), a web hook triggers all builds again. Now going through organizational and repository hooks to check what's happening.

The log says:

Sep 25, 2017 9:55:48 AM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent

Received PUSH for https://github.com/elifesciences/elife-jenkins-workflow-libs from 192.30.252.40 ⇒ http://ci--alfred.elifesciences.org:8080/github-webhook/

Sep 25, 2017 9:55:48 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

SCM changes detected in library-api-validator-php. Triggering #22

Sep 25, 2017 9:55:49 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

SCM changes detected in library-patterns-php. Triggering #434

Sep 25, 2017 9:55:49 AM INFO com.cloudbees.jenkins.GitHubPushTrigger$1 run

Since the web hook is organization-wide, I don't even know how does it understand that the repository is a global workflow library. I have gone through workspaces to delete the ...@libs folders to no avail.

Deleted all workspaces too, no change. The log of a triggered build continues to say:

 > git ls-remote -h git@github.com:elifesciences/elife-jenkins-workflow-libs.git # timeout=10
Found 1 remote heads on git@github.com:elifesciences/elife-jenkins-workflow-libs.git
[poll] Latest remote head revision on refs/heads/master is: 9f5110ead60321557cdc0e5394ec299d70983938
Using strategy: Default
[poll] Last Built Revision: Revision 966f0ebb926fbe6e2db5d2de68a86aec4ff86676 (refs/remotes/origin/develop)

and I don't know where it's getting that repository URL from.

I have found these references:

elife@ci--alfred:/var/lib/jenkins$ grep -r elife-jenkins-workflow-libs jobs/test-journal/builds/1056/build.xml                            
        <string>git@github.com:elifesciences/elife-jenkins-workflow-libs.git</string>
            <url>git@github.com:elifesciences/elife-jenkins-workflow-libs.git</url>

which are on a previous build rather than on the jobs definition, it is possible they are used to determine whether to trigger the job again. Will try to remove it on a test job.

To delete these references, I found out you have to run at least 1 (successful?) build for each job without the global pipeline library being defined in Global Settings. This seems to remove from the jobs/ metadata the reference to the external repository, so that when the Github hook triggers it is ignored. I am now running all builds once to clear all references before I am able to make changes to the library and push it to /workflowLibs.git.

Confirmed that poll SCM jobs are still triggered by every change to a shared library. What's the way forward on this? There are several duplicate JIRAs which describe this issue, but I don't see any open, active ones. Does this one need to be reopened, or does a new Major/Critical Bug need to be written?

dolphy possibly a new one (with repro details) cc abayer

As far as I can tell, the fix works. I noticed that existing jobs will need to be built at least once for this to take effect. This will "fix" the metadata recorded the SCM used. So when you upgrade to that version, the first next change to the shared library code might still triggers many jobs. Next changes will not.

ah thanks allan_burdajewicz!

cc dolphy

allan_burdajewicz michaelneale

You're right, and I'm wrong! I had based my comment solely off of seeing the jobs kick again, but never verified that they will continually kick. After the first time, they no longer trigger on a shared library commit.

Sorry for the hasty assumption, and thanks!

The global configuration option only for works for implicitly loaded libraries. If I include a library with the library step, I have to specify

library identifier: "MyLibrary", changelog: false

I would expect the default behavior

library "MyLibrary"

to be controlled by the global LibraryConfiguration option. As it is, I have to edit a large number of Jenkinsfiles to enable the new feature.

It seems like my expected behavior might have been intended originally. In the implementation in LibraryStep.java

public static class Execution extends AbstractSynchronousNonBlockingStepExecution<LoadedClasses> {
    @Override protected LoadedClasses run() throws Exception {
        // ...
        Boolean changelog = step.getChangelog(); // returns default value LibraryStep.changelog = true, if not provided
        // ...
        changelog = cfg.defaultedChangelogs(changelog); // returns changelog if != null, else global configuration option
        // ...
    }
}

a function is called that loads the global configuration value if the argument is null, but due to the default initialization, this statement only has effect if changelog = null is specified explicitly. However, a comment for LibraryStep.setChangelog states: default to including changes of the library in job recent changes.

Is this working as intended or should I open a PR for this?

mnsa do you mind opening that as a new feature JIRA for this and a PR that goes with it (if you want to) - and could be discussed there (or at least the ticket, where can be discussed). Changing default usually needs discussion but it seems reasonable in this case that the default is probably not right. 

With workflow-cps-global-lib 2.9 still seeing that jobs are re-built multiple times, with the option unchecked to include library changes in the config UI. I would agree that changing the default makes sense in this case because it is very unintuitive and can cause pretty large issues to your cluster throughput and stability, while if you wanted it and it gets turned off, you can easily turn it on.

From JENKINS-41497, "The global configuration option only for works for implicitly loaded libraries." For non-implicit libraries, is it expected that we have to set
changelog: false
..in every Jenkinsfile? If so, then the global setting should be a child checkbox under the implicit checkbox.

I opened issue JENKINS-48487 for this specific problem. It is actually only a problem of the the library step, using the '@Library' decorator, it works fine. For the library step you would have to specify changelog: false in every Jenkinsfile (and change it to true, if you should change your mind later on). Since it works for the decorator syntax, I don't think it should be a child checkbox of the implicit checkbox. I filed it as a bug because the behavior is different between the decorator and the step.

I see this issue once again with Jenkins 2.107.2, on 2.107.1 it is working just fine

This bug is still present on version 2.116. I found that the workaround mentioned here is the way to go for the moment. Thanks allan_burdajewicz!

Hello and sorry for being bit thick, but where is this "global configuration" located and changed?

I am trying to have implicitly loaded library for certain folder, but I cant figure how to stop the builds triggering continuously from shared lib commits.

Go to Manage Jenkins -> Configure System and there find the section "Global Pipeline Libraries"

"Dont trigger build a build on commit notifications" this setting for the library should stop the triggering?

This is what I already tested, and no matter if I let jobs trigger and build, they will trigger second, third and fourth time as well from commits to shared library.

lakeend please pay close attention to this comment and this other comment above.

I am confused as to what the "fix" is. I enabled the setting "Dont trigger build a build on commit notifications" for all our Jenkins jobs and they still get triggered whenever the global library SCM has a commit and this keeps happening after the first time. Do I need to remove the global library and re-add it again for this to work?

Seeing this exact issue on Jenkins 2.150.1, Build Pipeline 1.5.8. Have a global library set up in one repo/configured under Global Configuration, and a multibranch job that polls a different repo for changes to specific branches. Upon committing anything to any file in the global library repo, the multibranch job is triggered. I've read every thread I can find on the issue but cannot figure out how to not have jobs triggered when committing changes to the global library repo. It's basically unusable at this point, as I can't continue to trigger all jobs over and over. 

whatsdevops not sure if that is related to this issue: but I think that may be possible by using "when" and looking at the cause/trigger. So when global libraries are updated, the job will run, but terminate early (green) as it can skip the rest of the build. Kind of a work around. 

valide workaround for me was to add an additional behavior at the pipeline definition for the jenkinsfile or at the shared library repo definition, where to use ignore certain path with an inclusion of "ignoreRepo". no more additional builds. hope that bug get fixed or the option will disabled...

swtch3k another solution to your issue is to use "lightweight" checkout when defining the Jenkinsfile SCM.  I think it was intended behavior to allow polling on changes to the jenkinsfile scm - but lightweight will disable that (if you are using an SCM that supports it).

wbrode I do not see that the lightweight checkout actually solve this issue.

I do not understand why and how this issue is still open. this is a MAJOR problem with Jenkins....

wbrode as doronshai has already said, this unfortunately does not help to solve the problem...

Just would like to confirm, that I'm observing this undesirable behaviour too on Jenkins v2.161 and all the relevant pipeline plugins (up-to-date). All source code including Jenkinsfiles are in SVN, shared libs in Git/Bitbucket.

Nearly after month of migrating legacy builds to new pipelines and going through the loop of learning and refactoring my Jenkinsfiles to make them leaner and started using a [shared library|https://jenkins.io/doc/book/pipeline/shared-libraries/], each time I do a change in a shared lib, it triggers all the pipeline jobs. It's really nightmare.

I understand that there might be a need for this behaviour if a sort of part of a build script has been changed, to validate the rest of the code it is still buildable, OTOH if you're in a building stage of setting up your build system, this is incredible annoying. (For the same reason, I'm almost thinking to setup external repo just for the Jenkinsfiles to avoid unnecessary triggers of build jobs when modified Jenkinsfile in SCM. Having it inside the job configuration or using old fashioned freestyle jobs, exactly achieve this, but without additional benefits of using pipelines.)

So ideally, this behaviour would be configurable (like as it is now for Include @Library changes in job recent changes.)

MOVING TO GITLAB CI - THANKS FOR THE AMAZING SUPPORT

No matter how many new features being added to Jenkins, lack of support and issue fixing for such basic issues will eventually drove users away.

Goodbye Jenkins

I'm also affected by this issue running Jenkins ver. 2.176.2 with "Pipeline: Shared Groovy Libraries" version 2.14

I just recently switched to shared libs from internal workflowLibs.git. I added a global shared library using Subversion defined with "Load implicitly: true", "Allow default version to be overridden: true" and "Include @Library changes in job recent changes: true".
After the global library was used in the build for the first time, subsequent builds are now triggered whenever something changes in the global library.

I was not able to resolve this by simply deactivating "Include @Library changes in job recent changes". Still, builds were triggered for every commit in the shared lib. Only after a Jenkins restart ("Include @Library changes in job recent changes" is still false) the job is no longer triggered.

Why are polling and changelog bound to each other in shared libraries in the first place? In the "checkout" step these are two different options (or am I misreading something?).

In my case, the project that is triggered does NOT poll the SCM but is configured to only react on push notifications. So, this might be a different issue? Should I open a new bug report?

Even we are facing same issue , opened a ticket - https://issues.jenkins-ci.org/browse/JENKINS-52816

Is this still happening even after jobs are rebuilt at least once (see my comment above) ?

If still hitting the problem, https://issues.jenkins-ci.org/browse/JENKINS-61415 (workflow-job:2.37) may help as it does not require a rebuild anymore. (nor a restart of Jenkins)

in version 2.9 of workflow-cps-global-lib-plugin you added "allow excluding shared libraries from changelogs" as part of this commit https://github.com/jenkinsci/workflow-cps-global-lib-plugin/commit/64df779249ac69981a17f46e2698791fe95fa06d , but you never mentioned of how to do it in jenkins declarative pipeline?

well, this is the syntax of loading shared library into the pipeline

libraries {
  lib('some-library')
}

but I couldn't find any way to add the annotation of changelog=false in the jenkins declarative pipeline syntax. any idea how?

 and I couldn't find anything here: https://www.jenkins.io/doc/book/pipeline/shared-libraries/

some examples can be found here for jenkins scripted pipeline: https://stackoverflow.com/questions/46735908/jenkins-pipeline-global-pipeline-library-updates-show-up-in-jobs-list-of-rece
 

dnusbaum timja any chance that you can help here?

You can configure it in the global config (which is what we do) no idea about in the pipeline

Hi timja , thank you for your reply. I forgot to mention it but I already tried to configure it in the global config but it's not affect on my declarative pipeline which located in github organization. So in case that there are new commits in the shared library git repo I see those changes as part of the pipeline which can be confusing or affect the change set array that I use to detect what files has been changed 

dordor We have the same issue with declarative pipelines where I can't figure out how to turn off changelog. The global configuration doesn't seem to work. It's turned off but changelogs is still included for shared libraries.
We're using v2.17 of workflow-cps-global-lib-plugin.

I am facing similar issue. When I unselect 'Include @Library changes in job recent changes' and restart Jenkins, changelog does not show changes made to shared library but any change in shared library is still triggering the build.