-
Bug
-
Resolution: Fixed
-
Major
-
Powered by SuggestiMate -
support-core-2.63
Right now the plugin requires to have the master restarted otherwise (often?) it doesn't start to send bundles.
We want to verify if we could fix this issue (it's painful to have to restart an instance to install a plugin)
I suppose the issue could come from the instanciation/activation of the AsyncPeriodicWork task
Is there any good practices to follow ? Any sample of such AsyncPeriodicWork extensions which aren't requiring to restart the master ?
- is related to
-
JENKINS-59775 Support Core dynamic loading is failing to load Master Log Record component
-
- Closed
-
- links to
[JENKINS-59696] support-core and thus advisor require to restart the instance to be activated
In fact doing 3/ with the current master code and not the patched one works too
Should I conclude that to install a plugin without restarting the master the best strategy is to upload it manually instead of installing it from the UC?
As seen above and with allan_burdajewicz tests, the problem is perhaps in support-core and not in advisor ( JENKINS-59775 )
I did a lot of manual tests and the problem is always when support-core isn't yet installed. The problem isn't directly in advisor recampbell
Otherwise we will have to see with allan_burdajewicz if we can isolate/fix the issue in support-core.
The only thing I could try to do at the advisor level is to catch this error and ask to the administrator to restart the instance.
For now the user is seeing this generic error in the Advisor config panel:
And this warning monitor (note to myself: they should be unified ...)
Note, that as soon the support-core is installed but the instance not restarted the logs are polluted with these errors
Oct 16, 2019 8:49:05 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:08 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:11 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:14 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:17 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:20 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:23 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:26 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
I tested with the oldest version supported by the plugin (2.138.4) but as discussed with danielbeckI also tested the most recent LTS (2.190.1) and the behavior is the same.
I think we should reclassify this issue as a support-core.
Does it makes sense for all of you ?
Tested the following which confirms that Support Core is the culprit:
- Use fresh install of LTS 2.138.4
- Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough)
- Restart Jenkins
- Install Support Core from the Plugin Manager
- The exception shows up
Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the $JENKINS_URL/support aggressively (but manually ).
I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the SlowRequestChecker) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of ContentFilters to be loaded twice (In the ExtensionList, the 2 instances of ContentFilters are the same object).
Maybe SlowRequestChecker starts too soon.
Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the DeadlockTrackChecker because of the call from DeadlockTrackChecker. Which could be avoid there. But another call that could be causing this is the call to ExtensionList.lookupSingleton that can cause the ExtensionList to be created is it does not exist yet:
return ExtensionList.lookupSingleton(ContentFilters.class);
A fix I have been testing is to do this instead:
return GlobalConfiguration.all().get(ContentFilters.class);
I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component.
****
I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d
I noticed that when the issue happens:
- the extension gets created when loading the DeadlockTrackChecker going through: https://github.com/Dohbedoh/jenkins/blob/JENKINS-59696/core/src/main/java/jenkins/model/Jenkins.java#L2641.
- then the CompleteBatchJob refreshes the extensions and add it again through the ExtensionList#refresh
I have attached logs in extension-loading-with-issue.log
If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.
I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here.
With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical
pierrebtz also found something similar done by stephenconnolly a year ago:
https://github.com/jenkinsci/metrics-plugin/blame/bb4a572004bb708136a57d2d359a930c934786b1/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L708-L721
https://issues.jenkins-ci.org/browse/JENKINS-28983
https://issues.jenkins-ci.org/browse/JENKINS-52072
But my tests are with 2.138 thus we should have the required fix in core
And strangely I find ~ no plugin using @Extension(dynamicLoadable = YesNoMaybe.YES)