Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66017

OOM: unable to create new native thread (many RxNewThreadScheduler threads)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core, kubernetes-plugin
    • Labels:
      None
    • Environment:
      - Jenkins 2.289.1
      - k8s v1.18.3 deployment with jenkins helm-chart 3.3.22
    • Similar Issues:

      Description

      After about a week of uptime our Jenkins instance stops working with java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached.

      After installing the monitoring plugin and looking at the threads regularly we see instances of RxNewThreadScheduler-X accumulating in TIMED_WAITING. Those seem to be the cause of the problem.

      Unfortunately I cannot pinpoint what component or plugin is causing those threads to spawn. The instance is exclusively running pipelines that spawn kubernetes nodes via the kubernetes plugin.

      RxNewThreadScheduler-19
      "RxNewThreadScheduler-19" Id=2990 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@684980b7
      	at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@684980b7
      	at java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
      	at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
      	at java.base@11.0.11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
      	at java.base@11.0.11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
      	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
      	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
      	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)
      

      A list of currently installed plugins is attached.

        Attachments

        1. common.txt
          0.9 kB
        2. common-plugins.txt
          1 kB
        3. plugins.txt
          2 kB

          Issue Links

            Activity

            Hide
            chewplastic Jesse Jordan added a comment -

            Karthikeya/Joe Athman/Philipp Moeller after looking at your list of plugins, I noticed we also use influxdb and found this bug https://issues.jenkins.io/browse/JENKINS-65622

            Show
            chewplastic Jesse Jordan added a comment - Karthikeya / Joe Athman / Philipp Moeller  after looking at your list of plugins, I noticed we also use influxdb and found this bug  https://issues.jenkins.io/browse/JENKINS-65622
            Hide
            jjathman Joe Athman added a comment -

            Jesse Jordan - WOW! That is a great catch! The new InfluxDB Java Client API is a really bad design. I think many people will have this exact same problem. 

            Show
            jjathman Joe Athman added a comment - Jesse Jordan  - WOW! That is a great catch! The new InfluxDB Java Client API is a really bad design. I think many people will have this exact same problem. 
            Hide
            jakub_bednar Jakub added a comment -

            Hi All,

            I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121

            Regards

            Show
            jakub_bednar Jakub added a comment - Hi All, I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121 Regards
            Hide
            chewplastic Jesse Jordan added a comment -

            Thanks y'all, what a delight

            Show
            chewplastic Jesse Jordan added a comment - Thanks y'all, what a delight
            Hide
            pmr Philipp Moeller added a comment -

            Jakub Thank you! I can confirm that the InfluxDB Plugin 3.0.2 releases fixes our issue.

            Show
            pmr Philipp Moeller added a comment - Jakub Thank you! I can confirm that the InfluxDB Plugin 3.0.2 releases fixes our issue.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              pmr Philipp Moeller
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: