Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14336

Too many open files upon HTTP listener init or shutdown

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Jenkins 1.472 on RHEL 5 x86_64

      Trying to navigate to Jenkins web site, browser keeps loading and loading. When checked on Jenkins console it showed that Jenkins failed with the following:

      Jul 6, 2012 1:49:28 PM winstone.Logger logInternal
      SEVERE: Error during HTTP listener init or shutdown
      java.net.SocketException: Too many open files
              at java.net.PlainSocketImpl.socketAccept(Native Method)
              at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
              at java.net.ServerSocket.implAccept(ServerSocket.java:462)
              at java.net.ServerSocket.accept(ServerSocket.java:430)
              at winstone.HttpListener.run(HttpListener.java:136)
              at java.lang.Thread.run(Thread.java:662)
      

      I have verified open files and it seems that unreasonable large number of descriptors is open to various jar files, with jenkins-core being the leader:

      $ ls -la /proc/26867/fd | grep jar | wc -l
      876
      $ ls -la /proc/26867/fd | grep jenkins-core | wc -l
      462
      $ ls -la /proc/26867/fd | grep stapler | wc -l
      173

          [JENKINS-14336] Too many open files upon HTTP listener init or shutdown

          Jeremiah Roth added a comment -

          Also seeing this on 1.466.2, running in a tomcat6 container on Ubuntu 12.04. However, it happens without the HTTP init/shutdown trigger.

          We have not been able to determine what the trigger is, but I do know that when it happens, our number of open file descriptors jumps quickly (from about 1100 normally, to the terminal point of 4097 in just over an hour). ulimit set to: tomcat6 soft nofile 10000000 in /etc/security/limits.conf

          When this increase in java FD's occurs, it all comes from duplicate instances (thousands) of jenkins-core-1.466.2.jar.

          catalina.out doesn't indicate any problems until the "Too many open files" error starts showing up at about 3900 open file descriptors.

          Jeremiah Roth added a comment - Also seeing this on 1.466.2, running in a tomcat6 container on Ubuntu 12.04. However, it happens without the HTTP init/shutdown trigger. We have not been able to determine what the trigger is, but I do know that when it happens, our number of open file descriptors jumps quickly (from about 1100 normally, to the terminal point of 4097 in just over an hour). ulimit set to: tomcat6 soft nofile 10000000 in /etc/security/limits.conf When this increase in java FD's occurs, it all comes from duplicate instances (thousands) of jenkins-core-1.466.2.jar. catalina.out doesn't indicate any problems until the "Too many open files" error starts showing up at about 3900 open file descriptors.

          Peter Kline added a comment -

          Same here for us, Jenkins 1.500 on Centos 5.8 Java 1.6.38
          Only solution is for us to restart Jenkins.

          Peter Kline added a comment - Same here for us, Jenkins 1.500 on Centos 5.8 Java 1.6.38 Only solution is for us to restart Jenkins.

          xoan vilas added a comment -

          We do see the same running jenkins 1.500 on Red Hat Enterprise Linux Server release 5.6 (Tikanga) and java 1.4.2

          Attached the jenkins log with the list of open files (most of then related to adjunct stuff from jenkins-core) and a screenshoot.

          It ramps up in less than one hour usually after a garbage collection and at random times without an easy pattern to identify. Around 2 times a week at any time day or night with low number of sessions as with high

          xoan vilas added a comment - We do see the same running jenkins 1.500 on Red Hat Enterprise Linux Server release 5.6 (Tikanga) and java 1.4.2 Attached the jenkins log with the list of open files (most of then related to adjunct stuff from jenkins-core) and a screenshoot. It ramps up in less than one hour usually after a garbage collection and at random times without an easy pattern to identify. Around 2 times a week at any time day or night with low number of sessions as with high

          Liya Katz added a comment -

          same with Jenkins 1.505 on Centos 5.6 x86_64

          Liya Katz added a comment - same with Jenkins 1.505 on Centos 5.6 x86_64

          Jeremiah Roth added a comment -

          We've taken to scheduling therapeutic tomcat restarts every couple weeks so as to not have Jenkins crash unexpectedly. We're discussing upgrading from openjdk-6 (1.6.0_24) to openjdk-7 in case it's a java issue. I'd be interested to know if anyone is already using openjdk-7 and still has this problem.

          Jeremiah Roth added a comment - We've taken to scheduling therapeutic tomcat restarts every couple weeks so as to not have Jenkins crash unexpectedly. We're discussing upgrading from openjdk-6 (1.6.0_24) to openjdk-7 in case it's a java issue. I'd be interested to know if anyone is already using openjdk-7 and still has this problem.

          To increase the maximum file descriptors, adjust ulimit. In a large deployment of Jenkins, 1024 is too small.

          See http://stackoverflow.com/questions/11017402/increase-ulimit-for-of-file-descriptors for example.

          If you think there's a file descriptor leak, please open separate tickets. See https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error for the information we need to track down the problem.

          Kohsuke Kawaguchi added a comment - To increase the maximum file descriptors, adjust ulimit. In a large deployment of Jenkins, 1024 is too small. See http://stackoverflow.com/questions/11017402/increase-ulimit-for-of-file-descriptors for example. If you think there's a file descriptor leak, please open separate tickets. See https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+too+many+open+files+error for the information we need to track down the problem.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/02e00449e412fa3d18d9c4a04a670eaad899ec91
          Log:
          [FIXED JENKINS-14336]

          Integrating the stapler fix.

          Compare: https://github.com/jenkinsci/jenkins/compare/793b6826e567...02e00449e412

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml http://jenkins-ci.org/commit/jenkins/02e00449e412fa3d18d9c4a04a670eaad899ec91 Log: [FIXED JENKINS-14336] Integrating the stapler fix. Compare: https://github.com/jenkinsci/jenkins/compare/793b6826e567...02e00449e412

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2942
          [FIXED JENKINS-14336] (Revision 02e00449e412fa3d18d9c4a04a670eaad899ec91)

          Result = UNSTABLE
          kohsuke : 02e00449e412fa3d18d9c4a04a670eaad899ec91
          Files :

          • core/pom.xml
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2942 [FIXED JENKINS-14336] (Revision 02e00449e412fa3d18d9c4a04a670eaad899ec91) Result = UNSTABLE kohsuke : 02e00449e412fa3d18d9c4a04a670eaad899ec91 Files : core/pom.xml changelog.html

          Jesse Glick added a comment - - edited

          Verified fixed. To reproduce, just start Jenkins, enable auto refresh, and wait for a refresh to occur; ls -l /proc/$pid/fd|grep jenkins-core|wc -l will increase by 2 (until the next garbage collection).

          Jesse Glick added a comment - - edited Verified fixed. To reproduce, just start Jenkins, enable auto refresh, and wait for a refresh to occur; ls -l /proc/$pid/fd|grep jenkins-core|wc -l will increase by 2 (until the next garbage collection).

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          http://jenkins-ci.org/commit/jenkins/0485bfd44169bd00562327860b2848f8657c7944
          Log:
          [FIXED JENKINS-14336]

          Integrating the stapler fix.

          (cherry picked from commit 02e00449e412fa3d18d9c4a04a670eaad899ec91)

          Conflicts:
          changelog.html
          core/pom.xml

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: http://jenkins-ci.org/commit/jenkins/0485bfd44169bd00562327860b2848f8657c7944 Log: [FIXED JENKINS-14336] Integrating the stapler fix. (cherry picked from commit 02e00449e412fa3d18d9c4a04a670eaad899ec91) Conflicts: changelog.html core/pom.xml

            kohsuke Kohsuke Kawaguchi
            raspy Krzysztof Malinowski
            Votes:
            13 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: