Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59696

support-core and thus advisor require to restart the instance to be activated

    XMLWordPrintable

Details

    • support-core-2.63

    Description

      Right now the plugin requires to have the master restarted otherwise (often?) it doesn't start to send bundles.

      We want to verify if we could fix this issue (it's painful to have to restart an instance to install a plugin)

       

      I suppose the issue could come from the instanciation/activation of the AsyncPeriodicWork task

      https://github.com/jenkinsci/cloudbees-jenkins-advisor-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/advisor/BundleUpload.java

      Is there any good practices to follow ? Any sample of such AsyncPeriodicWork extensions which aren't requiring to restart the master ?

      Attachments

        Issue Links

          Activity

            Tested the following which confirms that Support Core is the culprit:

            • Use fresh install of LTS 2.138.4
            • Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough)
            • Restart Jenkins
            • Install Support Core from the Plugin Manager
            • The exception shows up
            allan_burdajewicz Allan BURDAJEWICZ added a comment - Tested the following which confirms that Support Core is the culprit: Use fresh install of LTS 2.138.4 Installed all Support Core dependencies from the Plugin Manager (installing Metrics plugin and Credentials plugin is enough) Restart Jenkins Install Support Core from the Plugin Manager The exception shows up
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the $JENKINS_URL/support aggressively (but manually ).

            I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the SlowRequestChecker) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of ContentFilters to be loaded twice (In the ExtensionList, the 2 instances of ContentFilters are the same object).

            Maybe SlowRequestChecker starts too soon.

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited Can reproduce in core 2.201 but not consistently. Sometimes it happens, sometimes no. To reproduce it consistently, I launch the plugin install and refresh the main page or the $JENKINS_URL/support aggressively (but manually ). I wonder if there is a race condition somewhere. Maybe one of the Support Core component (like the SlowRequestChecker ) starts before the PluginManager has finished to load the plugin dynamically, which causes the same extension of ContentFilters to be loaded twice (In the ExtensionList, the 2 instances of ContentFilters are the same object). Maybe SlowRequestChecker starts too soon.
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the DeadlockTrackChecker because of the call from DeadlockTrackChecker. Which could be avoid there. But another call that could be causing this is the call to ExtensionList.lookupSingleton that can cause the ExtensionList to be created is it does not exist yet:

            return ExtensionList.lookupSingleton(ContentFilters.class);
            

            A fix I have been testing is to do this instead:

            return GlobalConfiguration.all().get(ContentFilters.class);
            

            I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component.

            ****

            I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d

            I noticed that when the issue happens:

            I have attached logs in extension-loading-with-issue.log

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited Adding fine logging in ExtensionList, I found out that the extensions may be added twice when loading the DeadlockTrackChecker because of the call from DeadlockTrackChecker . Which could be avoid there. But another call that could be causing this is the call to ExtensionList.lookupSingleton that can cause the ExtensionList to be created is it does not exist yet: return ExtensionList.lookupSingleton(ContentFilters.class); A fix I have been testing is to do this instead: return GlobalConfiguration.all().get(ContentFilters.class); I have not been able to reproduce the issue using this method. This method goes through a different code path to get the component. **** I have added logging to the ExtensionsList class in this branch: https://github.com/Dohbedoh/jenkins/commit/0bf0178e43b8b8b42c82ba0938b4c53c2ead0c6d I noticed that when the issue happens: the extension gets created when loading the DeadlockTrackChecker going through: https://github.com/Dohbedoh/jenkins/blob/JENKINS-59696/core/src/main/java/jenkins/model/Jenkins.java#L2641 . then the CompleteBatchJob refreshes the extensions and add it again through the ExtensionList#refresh I have attached logs in extension-loading-with-issue.log

            If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.

             

             

            aheritier Arnaud Héritier added a comment - If you are facing this issue and cannot restart your instance, disable the logger which is generating these logs up until you can restart.    

            I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here.
            With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical

            aheritier Arnaud Héritier added a comment - I confirm. I tested with 2.138.4 and 2.190.2 and the problem isn't here. With 2.190.2 I hit JENKINS-60073 / JENKINS-59775 which is annoying but not critical

            People

              allan_burdajewicz Allan BURDAJEWICZ
              aheritier Arnaud Héritier
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: