Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57725

Proc.executor inherits ContextClassLoader dynamically

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • Jenkins 2.181

      The threads in the thread pool used for Proc.joinWithTimeout inherit their context class loader from the thread that calls Proc.joinWithTimeout, which seems undesirable. While looking into JENKINS-50223, I found a case where one of the threads in the thread pool was holding onto a CleanGroovyClassLoader class loader from workflow-cps (see the attached screenshot), which is definitely undesirable.

      We should wrap this thread pool in ClassLoaderSanityThreadFactory to prevent this kind of issue from happening.

          [JENKINS-57725] Proc.executor inherits ContextClassLoader dynamically

          Kyle added a comment - - edited

          This was mentioned in the LTS changelog for v2.176.2, but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.

           

           A java heap dump shows that  4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.


           

           Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

          Kyle added a comment - - edited This was mentioned in the  LTS changelog for v2.176.2 , but I'm experiencing something related.  My Jenkins system with 20 executors will creep towards 20k Tasks (processes + threads) before it hits the systemd imposed DefaultTasksMax limit of 19660 on my system.    A java heap dump shows that  4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader-> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader .    Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with. Ubuntu 18.04

          Kyle added a comment - - edited

          Additional plot from netdata tasks section showing the task plot skyrocket before derailing the Jenkins process and making it unstable/undefined when the systemd CGroup imposes the TasksMax limit:

          SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
          java.lang.OutOfMemoryError: unable to create new native thread
          

          Kyle added a comment - - edited Additional plot from netdata tasks section showing the task plot skyrocket before derailing the Jenkins process and making it unstable/undefined when the systemd CGroup imposes the TasksMax limit: SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed java.lang.OutOfMemoryError: unable to create new native thread

          Devin Nusbaum added a comment -

          2bluesc It looks like you have a distinct issue with two aspects: One, something is causing a bunch of timer threads to be created and not cleaned up, and two, those timer threads are inheriting their context class loader dynamically. The first issue is more serious, to investigate it I would search the jenkinsci organization on GitHub for all uses of java.util.Timer, cross-check that with the set of plugins you have installed, and then examine the ways that the plugins you have installed are using java.util.Timer based on the search. My guess is that something is repeatedly calling new Timer rather than storing a single timer somewhere and reusing it (maybe also better to switch to using an ExecutorService if that is the problem). Either way I'd go ahead and open a separate issue for your problem.

          Devin Nusbaum added a comment - 2bluesc It looks like you have a distinct issue with two aspects: One, something is causing a bunch of timer threads to be created and not cleaned up, and two, those timer threads are inheriting their context class loader dynamically. The first issue is more serious, to investigate it I would search the jenkinsci organization on GitHub for all uses of java.util.Timer , cross-check that with the set of plugins you have installed, and then examine the ways that the plugins you have installed are using java.util.Timer based on the search . My guess is that something is repeatedly calling new Timer rather than storing a single timer somewhere and reusing it (maybe also better to switch to using an ExecutorService if that is the problem). Either way I'd go ahead and open a separate issue for your problem.

          Kyle added a comment -

          dnusbaum thanks for the response, I've created a new issue @ JENKINS-58684 as you recommended and included all my plugins in the list. All of the plugins are pretty mainstream.

          Kyle added a comment - dnusbaum thanks for the response, I've created a new issue @ JENKINS-58684 as you recommended and included all my plugins in the list. All of the plugins are pretty mainstream.

            dnusbaum Devin Nusbaum
            dnusbaum Devin Nusbaum
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: