Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-4513

Hudson opens too many files

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • mercurial-plugin
    • None
    • Platform: PC, OS: Linux

    Description

      hudson@deadlock:/hudson$ lsof | grep hudson | wc
      72431 599541 7185484
      hudson@deadlock:/hudson$ lsof | grep hudson | grep pipe| wc
      47273 378184 4018205

      and:
      SEVERE: Failed to record SCM polling
      java.io.FileNotFoundException: /hudson/workdir/jobs/trunk/scm-polling.log (Too
      many open files)
      at java.io.FileOutputStream.open(Native Method)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
      at hudson.util.StreamTaskListener.<init>(StreamTaskListener.java:68)
      at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:311)
      at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:344)
      at
      hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:114)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

      server is running at http://deadlock.netbeans.org/hudson/

      Attachments

        Issue Links

          is blocking

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-4461 Hg polling can hang indefinitely

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            mzlamal mzlamal added a comment -

            .

            mzlamal mzlamal added a comment - .
            jglick Jesse Glick added a comment -

            First of all, each thread is listed separately. Currently running at around 70
            threads, so for example sezpoz-1.4.jar is listed twice per thread:

            java 19888 hudson mem REG 253,0 28546 11292071
            /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar
            java 19888 hudson 73r REG 253,0 28546 11292071
            /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar

            for a total of around 158 usages of this file.

            That said, the large number of open pipes seems odd. I don't know what code in
            Hudson would open a pipe.

            Short-term fix may simply be to increase the number of open files permitted for
            processes on the machine.

            jglick Jesse Glick added a comment - First of all, each thread is listed separately. Currently running at around 70 threads, so for example sezpoz-1.4.jar is listed twice per thread: java 19888 hudson mem REG 253,0 28546 11292071 /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar java 19888 hudson 73r REG 253,0 28546 11292071 /hudson/workdir/war/WEB-INF/lib/sezpoz-1.4.jar for a total of around 158 usages of this file. That said, the large number of open pipes seems odd. I don't know what code in Hudson would open a pipe. Short-term fix may simply be to increase the number of open files permitted for processes on the machine.
            jglick Jesse Glick added a comment -

            I just had to restart the server an hour ago, so whatever the problem is it
            isn't going away.

            The recent Mercurial plugin upgrade might be related, but I don't see the
            connection. Is there some way to tell what these "pipes" are? E.g.

            java 11140 hudson 167w FIFO 0,6 513046814
            pipe

            jglick Jesse Glick added a comment - I just had to restart the server an hour ago, so whatever the problem is it isn't going away. The recent Mercurial plugin upgrade might be related, but I don't see the connection. Is there some way to tell what these "pipes" are? E.g. java 11140 hudson 167w FIFO 0,6 513046814 pipe
            jglick Jesse Glick added a comment -

            I think it's a memory leak in java.lang.Process objects or something similar.

            jglick Jesse Glick added a comment - I think it's a memory leak in java.lang.Process objects or something similar.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in hudson
            User: : jglick
            Path:
            trunk/hudson/plugins/mercurial/src/main/java/hudson/plugins/mercurial/MercurialSCM.java
            http://fisheye4.cenqua.com/changelog/hudson/?cs=22014
            Log:
            [FIXED JENKINS-4513] Improper implementation of timeout (JENKINS-4461).

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in hudson User: : jglick Path: trunk/hudson/plugins/mercurial/src/main/java/hudson/plugins/mercurial/MercurialSCM.java http://fisheye4.cenqua.com/changelog/hudson/?cs=22014 Log: [FIXED JENKINS-4513] Improper implementation of timeout ( JENKINS-4461 ).
            kohsuke Kohsuke Kawaguchi added a comment -

            Somehow daemon failed to mark this issue as fixed, despite the commit message.

            kohsuke Kohsuke Kawaguchi added a comment - Somehow daemon failed to mark this issue as fixed, despite the commit message.

            People

              jglick Jesse Glick
              mzlamal mzlamal
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: