Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-4513

Hudson opens too many files

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • mercurial-plugin
    • None
    • Platform: PC, OS: Linux

      hudson@deadlock:/hudson$ lsof | grep hudson | wc
      72431 599541 7185484
      hudson@deadlock:/hudson$ lsof | grep hudson | grep pipe| wc
      47273 378184 4018205

      and:
      SEVERE: Failed to record SCM polling
      java.io.FileNotFoundException: /hudson/workdir/jobs/trunk/scm-polling.log (Too
      many open files)
      at java.io.FileOutputStream.open(Native Method)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
      at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:68)
      at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:311)
      at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:344)
      at
      hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:114)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

      server is running at http://deadlock.netbeans.org/hudson/

          [JENKINS-4513] Hudson opens too many files

          mzlamal added a comment -

          .

          mzlamal added a comment - .

          Jesse Glick added a comment -

          First of all, each thread is listed separately. Currently running at around 70
          threads, so for example sezpoz-1.4.jar is listed twice per thread:

          java 19888 hudson mem REG 253,0 28546 11292071
          /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar
          java 19888 hudson 73r REG 253,0 28546 11292071
          /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar

          for a total of around 158 usages of this file.

          That said, the large number of open pipes seems odd. I don't know what code in
          Hudson would open a pipe.

          Short-term fix may simply be to increase the number of open files permitted for
          processes on the machine.

          Jesse Glick added a comment - First of all, each thread is listed separately. Currently running at around 70 threads, so for example sezpoz-1.4.jar is listed twice per thread: java 19888 hudson mem REG 253,0 28546 11292071 /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar java 19888 hudson 73r REG 253,0 28546 11292071 /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar for a total of around 158 usages of this file. That said, the large number of open pipes seems odd. I don't know what code in Hudson would open a pipe. Short-term fix may simply be to increase the number of open files permitted for processes on the machine.

          Jesse Glick added a comment -

          I just had to restart the server an hour ago, so whatever the problem is it
          isn't going away.

          The recent Mercurial plugin upgrade might be related, but I don't see the
          connection. Is there some way to tell what these "pipes" are? E.g.

          java 11140 hudson 167w FIFO 0,6 513046814
          pipe

          Jesse Glick added a comment - I just had to restart the server an hour ago, so whatever the problem is it isn't going away. The recent Mercurial plugin upgrade might be related, but I don't see the connection. Is there some way to tell what these "pipes" are? E.g. java 11140 hudson 167w FIFO 0,6 513046814 pipe

          Jesse Glick added a comment -

          I think it's a memory leak in java.lang.Process objects or something similar.

          Jesse Glick added a comment - I think it's a memory leak in java.lang.Process objects or something similar.

          SCM/JIRA link daemon added a comment -

          Code changed in hudson
          User: : jglick
          Path:
          trunk/hudson/plugins/mercurial/src/main/java/hudson/plugins/mercurial/MercurialSCM.java
          http://fisheye4.cenqua.com/changelog/hudson/?cs=22014
          Log:
          [FIXED JENKINS-4513] Improper implementation of timeout (JENKINS-4461).

          SCM/JIRA link daemon added a comment - Code changed in hudson User: : jglick Path: trunk/hudson/plugins/mercurial/src/main/java/hudson/plugins/mercurial/MercurialSCM.java http://fisheye4.cenqua.com/changelog/hudson/?cs=22014 Log: [FIXED JENKINS-4513] Improper implementation of timeout ( JENKINS-4461 ).

          Kohsuke Kawaguchi added a comment -

          Somehow daemon failed to mark this issue as fixed, despite the commit message.

          Kohsuke Kawaguchi added a comment - Somehow daemon failed to mark this issue as fixed, despite the commit message.

            jglick Jesse Glick
            mzlamal mzlamal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: