Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18524

Too many open files caused Jenkins crashed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • other
    • None

      After Jenkins running for a long time, it suddenly stuck. We checked the log and found too many socket was opened and the number of opened files comes 65535. So it failed to accept new request. Could you please help to check why Jenkins opened so many socket and can't close it?
      We have tried to use gc to collect garbage memory, but it can't close these socket. So this should be a bug of Jenkins.
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/KillZombieProcess
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:12 AM hudson.model.Executor run
      SEVERE: Executor threw an exception
      hudson.util.IOException2: Failed to create a temporary file in /proj/RBS_IoV/.hudson/jobs/W12A_CLA_R6_CI_testExec_Trigger_Job
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:67)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:54)
      at hudson.util.TextFile.write(TextFile.java:78)
      at hudson.model.Job.saveNextBuildNumber(Job.java:234)
      at hudson.model.Job.assignBuildNumber(Job.java:291)
      at hudson.model.Run.<init>(Run.java:256)
      at hudson.model.AbstractBuild.<init>(AbstractBuild.java:158)
      at hudson.model.Build.<init>(Build.java:92)
      at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:37)
      at sun.reflect.GeneratedConstructorAccessor100.newInstance(Unknown Source)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:982)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1197)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:136)
      at hudson.model.Executor.run(Executor.java:211)
      Caused by: java.io.IOException: Too many open files
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.checkAndCreate(File.java:1704)
      at java.io.File.createTempFile(File.java:1792)
      at hudson.util.AtomicFileWriter.<init>(AtomicFileWriter.java:65)
      ... 15 more
      Jun 13, 2013 10:21:13 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
      SEVERE: Socket accept failed
      java.net.SocketException: Too many open files

        1. jstack.log
          0.7 kB
        2. lsof.log
          8.41 MB
        3. tomcat.log
          6 kB
        4. ulimit.log
          0.2 kB

          [JENKINS-18524] Too many open files caused Jenkins crashed

          martin naughton added a comment -

          did you investigate what the sockets might be doing?

          martin naughton added a comment - did you investigate what the sockets might be doing?

          Oleg Nenashev added a comment - - edited

          Maybe, the issue duplicates https://issues.jenkins-ci.org/browse/JENKINS-20051
          In this case, LDAP sockets hang due to LDAP server overloading.

          Oleg Nenashev added a comment - - edited Maybe, the issue duplicates https://issues.jenkins-ci.org/browse/JENKINS-20051 In this case, LDAP sockets hang due to LDAP server overloading.

          Daniel Beck added a comment -

          Are you using the Jenkins CLI (e.g. JENKINS-23572)? Does this issue also occur on current Jenkins versions?

          Daniel Beck added a comment - Are you using the Jenkins CLI (e.g. JENKINS-23572 )? Does this issue also occur on current Jenkins versions?

          Youcai Han added a comment -

          Yes, we used JCLI. But we haven't found this issue for about 1 year with the same Jenkins version 1.480.1.

          Youcai Han added a comment - Yes, we used JCLI. But we haven't found this issue for about 1 year with the same Jenkins version 1.480.1.

          Daniel Beck added a comment -

          eyouhan To clarify, you have not experienced the issue reported here for one year?

          If so, I suggest resolving as Cannot Reproduce. You can file a new issue anytime it occurs again with updated logs and config information...

          Daniel Beck added a comment - eyouhan To clarify, you have not experienced the issue reported here for one year? If so, I suggest resolving as Cannot Reproduce. You can file a new issue anytime it occurs again with updated logs and config information...

          Youcai Han added a comment -

          Yes, please close it.

          Youcai Han added a comment - Yes, please close it.

          Daniel Beck added a comment -

          As discussed, resolving as Cannot Reproduce as the reporter did not experience the issue in a year.

          Daniel Beck added a comment - As discussed, resolving as Cannot Reproduce as the reporter did not experience the issue in a year.

            Unassigned Unassigned
            hannyjxb Youcai HAN
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: