Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66017

OOM: unable to create new native thread (many RxNewThreadScheduler threads)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core, kubernetes-plugin
    • None
    • - Jenkins 2.289.1
      - k8s v1.18.3 deployment with jenkins helm-chart 3.3.22

      After about a week of uptime our Jenkins instance stops working with java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached.

      After installing the monitoring plugin and looking at the threads regularly we see instances of RxNewThreadScheduler-X accumulating in TIMED_WAITING. Those seem to be the cause of the problem.

      Unfortunately I cannot pinpoint what component or plugin is causing those threads to spawn. The instance is exclusively running pipelines that spawn kubernetes nodes via the kubernetes plugin.

      RxNewThreadScheduler-19
"RxNewThreadScheduler-19" Id=2990 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@684980b7
	at java.base@11.0.11/jdk.internal.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@684980b7
	at java.base@11.0.11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
	at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
	at java.base@11.0.11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
	at java.base@11.0.11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
	at java.base@11.0.11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base@11.0.11/java.lang.Thread.run(Thread.java:829)

      A list of currently installed plugins is attached.

        1. common.txt
          0.9 kB
        2. common-plugins.txt
          1 kB
        3. plugins.txt
          2 kB

          [JENKINS-66017] OOM: unable to create new native thread (many RxNewThreadScheduler threads)

          Joe Athman added a comment -

          We seem to be hitting this same issue. 

          Joe Athman added a comment - We seem to be hitting this same issue. 

          Joe Athman added a comment -

          timja - for what it's worth, we seem to be having the same issue but do not have the kubernetes plugin installed. 

          Joe Athman added a comment - timja  - for what it's worth, we seem to be having the same issue but do not have the kubernetes plugin installed. 

          Philipp Moeller added a comment -

          jjathmanWhat's the overlap between your and our plugin list?

          Philipp Moeller added a comment - jjathman What's the overlap between your and our plugin list?

          Joe Athman added a comment -

          pmr it's a pretty long list, but quite a few of them are just standard plugins. Here's the common plugins between our installation and what you have common-plugins.txt

          Joe Athman added a comment - pmr  it's a pretty long list, but quite a few of them are just standard plugins. Here's the common plugins between our installation and what you have  common-plugins.txt

          Karthikeya added a comment -

          We're seeing the same issue, here's a list of common plugins between our installation and the above common list to narrow it down even further - common.txt

          Karthikeya added a comment - We're seeing the same issue, here's a list of common plugins between our installation and the above common list to narrow it down even further - common.txt

          Jesse Jordan added a comment -

          paranoidsp/jjathman/pmr after looking at your list of plugins, I noticed we also use influxdb and found this bug https://issues.jenkins.io/browse/JENKINS-65622

          Jesse Jordan added a comment - paranoidsp / jjathman / pmr  after looking at your list of plugins, I noticed we also use influxdb and found this bug  https://issues.jenkins.io/browse/JENKINS-65622

          Joe Athman added a comment -

          chewplastic - WOW! That is a great catch! The new InfluxDB Java Client API is a really bad design. I think many people will have this exact same problem. 

          Joe Athman added a comment - chewplastic  - WOW! That is a great catch! The new InfluxDB Java Client API is a really bad design. I think many people will have this exact same problem. 

          Jakub added a comment -

          Hi All,

          I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121

          Regards

          Jakub added a comment - Hi All, I've prepare the PR with fixes: https://github.com/jenkinsci/influxdb-plugin/pull/121 Regards

          Jesse Jordan added a comment -

          Thanks y'all, what a delight

          Jesse Jordan added a comment - Thanks y'all, what a delight

          Philipp Moeller added a comment -

          jakub_bednar Thank you! I can confirm that the InfluxDB Plugin 3.0.2 releases fixes our issue.

          Philipp Moeller added a comment - jakub_bednar Thank you! I can confirm that the InfluxDB Plugin 3.0.2 releases fixes our issue.

            Unassigned Unassigned
            pmr Philipp Moeller
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: