Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11355

CLI stops working. java.lang.OutOfMemoryError: unable to create new native thread

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • None

    Description

      I run frequent scripts on a node called "utils" using the groovy CLI to communicate with Jenkins. After a couple of hours, the CLI stops working. I've found several java.lang.OutOfMemoryError: unable to create new native thread in the log file. I've checked the memory and it uses 500 mb out of 1.5gb allocated. I searched the net and found that it could be caused by too many threads.

      I've done a thread dump and found that the node called "utils" has 1074 threads that look like this:

      ComThread for Thread-1022

      "ComThread for Thread-1022" Id=1497 Group=main RUNNABLE (in native)
      at com4j.Win32Lock.suspend0(Native Method)
      at com4j.Win32Lock.suspend(Win32Lock.java:33)
      at com4j.ComThread.run0(ComThread.java:135)
      at com4j.ComThread.run(ComThread.java:125)

      Finally, Jenkins crashes with out of memory. Attached is the thread dump.

      Any ideas? It started around the time when I finally understood how the groovy CLI scripts work and added a few of them.

      -------------------

      The groovy scripts I run are:

       
      Thread.getAllStackTraces().keySet().each() { 
              item ->
              if (item.getName().contains("SCM polling") && item.getName().contains("waiting for hudson.remoting")) { 
                      println "Interrupting thread " + item.getId(); 
                      item.interrupt() 
              }
      }
      
       
      def slaves = hudson.model.Hudson.instance.getNodes()
      for (slave in slaves) {
        def comp = slave.getComputer() 
        def executors = comp.getExecutors() 
        if (executors != null ) {
          for (executor in executors) {
            def workunit = executor.getCurrentWorkUnit() 
            if (workunit != null) {
              def subtask = workunit.work
              println  "Slavename:" + slave.getDisplayName() + ";Task:BUSY:" + subtask.getDisplayName()
            }
            else {
              println  "Slavename:" + slave.getDisplayName() + ";Task:IDLE"
            }
          }
        }
      }
      
       
      def slaves = hudson.model.Hudson.instance.getNodes()
      for (slave in slaves) {
        def comp = slave.getComputer() 
        if (comp.isOnline()) {
          println "Name:" + comp.getName() + "; Status: ONLINE; Hostname:" + comp.getHostName() + "; Idle:" + comp.countIdle()
        }
        else{
          println "Name:" + comp.getName() + "; Status: OFFLINE;  Hostname:NA" + "; Idle:0"
        }
      }
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mabahj Markus
              Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated: