Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59696

support-core and thus advisor require to restart the instance to be activated

    • support-core-2.63

      Right now the plugin requires to have the master restarted otherwise (often?) it doesn't start to send bundles.

      We want to verify if we could fix this issue (it's painful to have to restart an instance to install a plugin)

       

      I suppose the issue could come from the instanciation/activation of the AsyncPeriodicWork task

      https://github.com/jenkinsci/cloudbees-jenkins-advisor-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/advisor/BundleUpload.java

      Is there any good practices to follow ? Any sample of such AsyncPeriodicWork extensions which aren't requiring to restart the master ?

          [JENKINS-59696] support-core and thus advisor require to restart the instance to be activated

          pierrebtz also found something similar done by stephenconnolly a year ago:
          https://github.com/jenkinsci/metrics-plugin/blame/bb4a572004bb708136a57d2d359a930c934786b1/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L708-L721
          https://issues.jenkins-ci.org/browse/JENKINS-28983
          https://issues.jenkins-ci.org/browse/JENKINS-52072

          But my tests are with 2.138 thus we should have the required fix in core

          And strangely I find ~ no plugin using @Extension(dynamicLoadable = YesNoMaybe.YES)

          Arnaud Héritier added a comment - pierrebtz also found something similar done by stephenconnolly a year ago: https://github.com/jenkinsci/metrics-plugin/blame/bb4a572004bb708136a57d2d359a930c934786b1/src/main/java/jenkins/metrics/impl/JenkinsMetricProviderImpl.java#L708-L721 https://issues.jenkins-ci.org/browse/JENKINS-28983 https://issues.jenkins-ci.org/browse/JENKINS-52072 But my tests are with 2.138 thus we should have the required fix in core And strangely I find ~ no plugin using @Extension(dynamicLoadable = YesNoMaybe.YES)

          In fact doing 3/ with the current master code and not the patched one works too

          Should I conclude that to install a plugin without restarting the master the best strategy is to upload it manually instead of installing it from the UC?

          Arnaud Héritier added a comment - In fact doing 3/ with the current master code and not the patched one works too Should I conclude that to install a plugin without restarting the master the best strategy is to upload it manually instead of installing it from the UC?

          allan_burdajewicz could we discuss about this ?

          Arnaud Héritier added a comment - allan_burdajewicz could we discuss about this ?

          As seen above and with allan_burdajewicz tests, the problem is perhaps in support-core and not in advisor ( JENKINS-59775 ) 

          Arnaud Héritier added a comment - As seen above and with allan_burdajewicz tests, the problem is perhaps in support-core and not in advisor (  JENKINS-59775  ) 

          I did a lot of manual tests and the problem is always when support-core isn't yet installed. The problem isn't directly in advisor recampbell

          Otherwise we will have to see with allan_burdajewicz if we can isolate/fix the issue in support-core.

          The only thing I could try to do at the advisor level is to catch this error and ask to the administrator to restart the instance.

          For now the user is seeing this generic error in the Advisor config panel:

           

          And this warning monitor (note to myself: they should be unified ...)

          Note, that as soon the support-core is installed but the instance not restarted the logs are polluted with these errors

           

          Oct 16, 2019 8:49:05 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:08 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:11 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:14 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:17 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:20 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:23 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Oct 16, 2019 8:49:26 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed
          java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2
          	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450)
          	at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47)
          	at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80)
          	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
          	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          

           

          I tested with the oldest version supported by the plugin (2.138.4) but as discussed with danielbeckI also tested the most recent LTS (2.190.1) and the behavior is the same.

          I think we should reclassify this issue as a support-core. 

          Does it makes sense for all of you ?

           

          Arnaud Héritier added a comment - I did a lot of manual tests and the problem is always when support-core isn't yet installed. The problem isn't directly in advisor recampbell Otherwise we will have to see with allan_burdajewicz if we can isolate/fix the issue in support-core. The only thing I could try to do at the advisor level is to catch this error and ask to the administrator to restart the instance. For now the user is seeing this generic error in the Advisor config panel:   And this warning monitor (note to myself: they should be unified ...) Note, that as soon the support-core is installed but the instance not restarted the logs are polluted with these errors   Oct 16, 2019 8:49:05 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:08 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:11 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:14 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:17 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:20 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:23 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Oct 16, 2019 8:49:26 AM hudson.triggers.SafeTimerTask run SEVERE: Timer task com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker@2656884b failed java.lang.IllegalStateException: Expected 1 instance of com.cloudbees.jenkins.support.filter.ContentFilters but got 2 at hudson.ExtensionList.lookupSingleton(ExtensionList.java:450) at com.cloudbees.jenkins.support.filter.ContentFilters.get(ContentFilters.java:47) at com.cloudbees.jenkins.support.SupportPlugin.getContentFilter(SupportPlugin.java:402) at com.cloudbees.jenkins.support.slowrequest.SlowRequestChecker.doRun(SlowRequestChecker.java:80) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   I tested with the oldest version supported by the plugin (2.138.4) but as discussed with danielbeck I also tested the most recent LTS (2.190.1) and the behavior is the same. I think we should reclassify this issue as a support-core.  Does it makes sense for all of you ?  

          Tested the following which confirms that Support Core is the culprit:

          • Use fresh install of LTS 2.138.4
          • Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough)
          • Restart Jenkins
          • Install Support Core from the Plugin Manager
          • The exception shows up

          Allan BURDAJEWICZ added a comment - Tested the following which confirms that Support Core is the culprit: Use fresh install of LTS 2.138.4 Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough) Restart Jenkins Install Support Core from the Plugin Manager The exception shows up

          Allan BURDAJEWICZ added a comment - - edited

          Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the $JENKINS_URL/support aggressively (but manually ).

          I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the SlowRequestChecker) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of ContentFilters to be loaded twice (In the ExtensionList, the 2 instances of ContentFilters are the same object).

          Maybe SlowRequestChecker starts too soon.

          Allan BURDAJEWICZ added a comment - - edited Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the $JENKINS_URL/support aggressively (but manually ). I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the SlowRequestChecker ) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of ContentFilters to be loaded twice (In the ExtensionList, the 2 instances of ContentFilters are the same object). Maybe SlowRequestChecker starts too soon.

          Allan BURDAJEWICZ added a comment - - edited

          Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the DeadlockTrackChecker because of the call from DeadlockTrackChecker. Which could be avoid there. But another call that could be causing this is the call to ExtensionList.lookupSingleton that can cause the ExtensionList to be created is it does not exist yet:

          return ExtensionList.lookupSingleton(ContentFilters.class);
          

          A fix I have been testing is to do this instead:

          return GlobalConfiguration.all().get(ContentFilters.class);
          

          I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component.

          ****

          I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d

          I noticed that when the issue happens:

          I have attached logs in extension-loading-with-issue.log

          Allan BURDAJEWICZ added a comment - - edited Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the DeadlockTrackChecker because of the call from DeadlockTrackChecker . Which could be avoid there. But another call that could be causing this is the call to ExtensionList.lookupSingleton that can cause the ExtensionList to be created is it does not exist yet: return ExtensionList.lookupSingleton(ContentFilters.class); A fix I have been testing is to do this instead: return GlobalConfiguration.all().get(ContentFilters.class); I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component. **** I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d I noticed that when the issue happens: the extension gets created when loading the DeadlockTrackChecker going through: https://github.com/Dohbedoh/jenkins/blob/JENKINS-59696/core/src/main/java/jenkins/model/Jenkins.java#L2641 . then the CompleteBatchJob refreshes the extensions and add it again through the ExtensionList#refresh I have attached logs in extension-loading-with-issue.log

          If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.

           

           

          Arnaud Héritier added a comment - If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.    

          I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here.
          With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical

          Arnaud Héritier added a comment - I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here. With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical

            allan_burdajewicz Allan BURDAJEWICZ
            aheritier Arnaud Héritier
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: