So if I'm parsing some of the similar issues correctly, such as https://github.com/SonarSource/sonar-scanner-jenkins/pull/42

and https://github.com/jenkinsci/sonarqube-plugin/commit/20431dcd80cf34a674db1197e90e13a160865def, it looks like

the calling of save() in the default contstructor is the issue.

At first blush, it does seem unnecessary.  I'll test a bit with it removed, and if things look OK, will cut a new plugin version with

this change.

I'm also having this exact problem and I'm not sure why it's happening now. I have several deployments on OpenShift (3.6.0 & 3.6.1) recently and it worked at plugin version 0.1.23, but not my newly built Jenkins containers are failing with this same error at that SAME version. Could this be some sort of conflict with another plugin?

New information... I downgraded to a MUCH earlier set of plugins and then started upgrading plugins one at a time to see when things would break...

As weird as it sounds, I could upgrade the `kubernetes` plugin and the `openshift-(sync|login|pieline)` plugins to the latest with no issue. Then, I upgraded my `blueocean` and `blueocean-pipeline-editor` plugins to 1.3.0 and the error appeared! I'll try to dig in and discover what the connection might be and report back.

OK, I believe I found where things go sideways, but I have yet to be able to pin it down to a specific root cause. Upgrading BlueOcean to anything >= `1.2.0` causes the Guice dependency injections for `openshift-sync` to fail. I have performed a diff of blueocean 1.1.7 to 1.2.0 and discovered the following likely culprits:

1. Jackson Upgrade
2. Maven Upgrade
3. JODA Time implementation
4. Version mismatch in `org.apache:commons-lang` and `org.apache:commons-lang3`

I will open a corresponding ticket against BlueOcean and reference this issue.

Thanks for all the investigation Deven.

I have not had a chance to do any testing with it, but I can attach a non-release patch for openshift-sync that removes the save() call I referenced in

https://issues.jenkins-ci.org/browse/JENKINS-42276?focusedCommentId=316652&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-316652

Would you have time to try it out and see if it addresses things with the later BlueOcean ?

File attached ... see the icon above, or hit the link openshift-sync.hpi

thanks again

gmontero

Unfortunately, it's quite difficult to try out with our static Jenkins Ephemeral implementation. Unless the plugin is in the Jenkins plugin repository our S2I image build cannot pick it up.

Gotcha - no problem.

I've initiated some regression testing.  If it passes, I may be able to cut a new plugin version soon, but the waters are a bit muddy there since we are trying to lock

down our current release for the overall product.  I'll report back when I have an update.

potential work around with sync plugin currently being worked via https://github.com/openshift/jenkins-sync-plugin/pull/184

regression suite passed ... will move to testing with blueocean >= 1.2.0 next.

if all goes well, will craft change in https://github.com/jenkinsci/openshift-sync-plugin and cut new version.

PR https://github.com/openshift/jenkins-sync-plugin/pull/184 has merged and the commit picked to jenkinsci

I'll craft a new release of the plugin after Thanksgiving

gmontero infosec812 this plugin.txt will recreate the issue. At least in this configuration, it appears that the root cause is transitive dependencies not being met for various plugins. I'm not sure if georgestark had the same issue without seeing a complete log. See jenkins-7-s62nq.log.

I think the real issue is the NPE with the various watchers, as this badly breaks the sync plugins functions. Therefore, this issue is a bit of a red herring. I'll try to get a plugins.txt that removes this transitive dependency issue and then try openshift-sync.hpi

gmontero 

Using the latest plugins, including blue ocean 1.3.3 and ocp syc 0.1.32, I see the circular dependency issue, but I do not have any breaking functionality of sync plugin. See this branch and jenkins-11.log. The logs show a job syncing and executing correctly. 

It appears the .hpi provided here breaks sync functionality with latest blue ocean (1.3.3). Jobs and builds not syncing. See this branch, which has openshift-sync.hpi and removes sync from plugins.txt. Here is the jenkins-10.log 

Ignore the sonarqube noise. That is our config and I do not expect it to work in this example. If you would like to fully recreate this test env, see the README in the root directory of the examples repos. It will use ansible to apply a bunch of OCP templates, including the build configs with jenkins pipeline strategy.

and before I forget, to reproduce, please build with the plugin.txt provided. Do not update via the Jenkins UI. I think there is a bootstrap operation that causes this to occur when running from a cold start, but does occur when running from a plugin update to a running process.

Justin - took a look at jenkins-7-s62nq.log ; based on what I see there, your https://github.com/sherl0cks/labs-ci-cd/blob/jenkins-42276-recreate/s2i-config/jenkins-master/plugins.txt file is invalid.  The dependencies have

not been properly resolved.

How did you go about constructing that file?

When we construct the analogous file when building the openshift jenkins image, we go through a process of making sure the dependencies are at the latest minimum version possible, such that exceptions like

SEVERE: Failed Loading plugin Pipeline SCM API for Blue Ocean v1.2.0 (blueocean-pipeline-scm-api)
java.io.IOException: Pipeline SCM API for Blue Ocean v1.2.0 failed to load.
 - Branch API Plugin v2.0.10 is older than required. To fix, install v2.0.11 or later.
    at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:621)
    at hudson.PluginManager$2$1$1.run(PluginManager.java:517)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
    at jenkins.model.Jenkins$7.runTask(Jenkins.java:1089)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Do no occur.  See https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/install-plugins.sh#L217-L297 in particular, and the install-plugins.sh file in general.

Along those lines though, since I've had trouble reproducing the circular dependency, if https://issues.jenkins-ci.org/secure/ViewProfile.jspa?name=georgestark or

https://issues.jenkins-ci.org/secure/ViewProfile.jspa?name=infosec812 could provide the complete list of plugins, not just the subset they noted, that resulted in their

circular dependency stack traces, I could a) reproduce by building an image with those, and b) then apply our sync plugin patch that removes the redudant save call

and confirm / deny whether it addresses the issue.

gmontero I created the file by manually updating in the web UI and then running a Groovy script in the console to extract the list of plugins/versions.

gmontero see  https://github.com/sherl0cks/labs-ci-cd/tree/jenkins-42276-latest/s2i-config/jenkins-master and jenkins-11.log for a properly resolved dependency tree that shows the circular dependency. 

I originally thought that the issue here was the unresolved deps, but the above branch and logs shows it is not. 

infosec812   oops .... I think our comments passed in the ether

OK, yeah, that seems to be it.  I'll take the plugin.txt there and start investigating / verifying / etc.  Thanks!

hey infosec812 - I tried running a s2i build from that branch / dir (https://github.com/sherl0cks/labs-ci-cd/tree/jenkins-42276-latest/s2i-config/jenkins-master), but when I run the image, I still get the "SEVERE: Failed Loading ... " messages

and not the circular dependency when I `docker run ` the image.

From the branch / dir local repo, I ran both:  s2i build . openshift/jenkins-2-centos7:latest jira-test:latest 

and: s2i build . registry.access.redhat.com/openshift3/jenkins-2-rhel7:latest jira-test:latest 

Any obvious differences you see re: building the image with the circular dependency issue?

Or could you perhaps push the image you used to generate jenkins-11.log

to your account on docker.io (assuming you have one) ?

Just fyi its Justin Holmes pushing the changes right now. Deven and I are team mates.

Maybe a commit missing somewhere on my end? Im travelling right now, will send an update in a couple hours. Can you share the log here so I can compare versions?

Sorry for the wild goose chase Gabe. Jenkins plugins are the bane of my existance...

ah .. sorry for the confusion sherl0cks on who to tag

and no worries ... getting a reliable repro will ease our worry (though the stack trace on the circular dep certainly points to the save() call in the GlobalPluginConfiguration I've previously noted)

here's the lattest commit in the log when I cloned the branch you mentioned:

gmontero ~/go/src/github.com/openshift/labs-ci-cd/s2i-config/jenkins-master  (jenkins-42276-recreate)$ git log -1
commit 75cf51cb93fc1eda71f480db5c9a02983b50b52c
Author: Justin Holmes <jholmes@redhat.com>
Date:   Sun Dec 3 13:56:19 2017 -0700

    adding back plugins to get closer to original build

Gabe can you confirm you are on the correct branch (jenkins-42276-latest)?

Github shows a different latest commit
https://github.com/sherl0cks/labs-ci-cd/commits/jenkins-42276-latest

yep that was it ... had the recreate vs. latest branch .... have reproduced, about to test fix

I was able to successfully verify the fix using sherl0cks 's repro env.

My steps were, from my clone of https://github.com/sherl0cks/labs-ci-cd/tree/jenkins-42276-latest/s2i-config/jenkins-master

1) remove openshfit-sync from the plugins.txt

2) create a plugins dir off of the jenkins-master dir

3) copy the updated plugin to plugins/

4) ran s2i build . openshift/jenkins-2-centos7:latest jira-test:latest 

I then docker ran jira-test:latest and all was good.

That s2i command can extend the openshift jenkins rhel image as well.  But you have to name the file in the plugins directory

as "openshift-sync.jpi" prior to running the s2i command.  There are old, gory details about how jenkins overrides plugins that apply

here, along with differences in how the plugins get installed with the centos vs. rhel image.

With all that, as I alluded to before, I'll resolve this bug once I cut a new version of openshift-sync up at the jenkins update center.  Should be this week.

v0.9.1 of the sync plugin has the fix for this and is available at the jenkins update center

the openshift jenkins centos image is also available on docker hub

marking resolved

i  build jenkins master image  and  jenkins slave image  like  follow  steps.  but when i start deploying the jenkins master ,  same error  occured.  can  u  give some advices? Thanks

jenkins master  base image:   jenkins-2.32-rhel7

jenkins slave base  image:  rhel7

install on  master:  swarm plugin , gearman  plugin ,   zuul 

install on slave :  swarm plugin,  maven