I have a job that dynamically creates jobs using the CLI. I have noticed that since installing the job, which verifies the existing of jobs by calling 'get-job', Jenkins is leaking file descriptors. I am currently making around 40 calls per build, which runs on every CVS commit. I have a job setup to monitor the number of FD's in /proc/$jenkins_pid/fd. Calling garbage collection in the JVM doesn't release the FD's and thus the only cure is to restart Jenkins before the number of files reaches the Open file ulimit. I have set my ulimit to 65356 so I don't have to reboot so frequently. I restarted Jenkins at 7:49 this morning and the file descriptor count is currently at 6147 files; it's now 12:10 in the afternoon, so it's been steadily leaking FD's at approximately 1500 per hour.

          [JENKINS-23248] CLI calls are causing file descriptor leaks.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          src/main/java/hudson/remoting/SocketChannelStream.java
          http://jenkins-ci.org/commit/remoting/c90fc463923cc7f4a8907a0b352204f3d561cc55
          Log:
          [FIXED JENKINS-23248] Seeing strange "Transport endpoint is not connected" exception

          s.shutdownInput() fails with the following exception, even though s.isInputShutdown() is reporting false:

          java.net.SocketException: Transport endpoint is not connected
          at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
          at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:667)
          at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:378)
          at hudson.remoting.SocketChannelStream$1.close(SocketChannelStream.java:39)
          at sun.nio.ch.ChannelInputStream.close(ChannelInputStream.java:113)
          at javax.crypto.CipherInputStream.close(CipherInputStream.java:296)
          at java.io.BufferedInputStream.close(BufferedInputStream.java:468)
          at hudson.remoting.FlightRecorderInputStream.close(FlightRecorderInputStream.java:112)
          at hudson.remoting.ChunkedInputStream.close(ChunkedInputStream.java:102)
          at hudson.remoting.ChunkedCommandTransport.closeRead(ChunkedCommandTransport.java:50)
          at hudson.remoting.Channel.terminate(Channel.java:795)
          at hudson.remoting.Channel$CloseCommand.execute(Channel.java:951)
          at hudson.remoting.Channel$2.handle(Channel.java:475)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)

          This bug report may be related: http://bugs.java.com/view_bug.do?bug_id=4516760
          If we fail to call s.close(), a socket will leak, so swallowing this exception and have the code execute "s.close()"

          Compare: https://github.com/jenkinsci/remoting/compare/d1905acf329e...c90fc463923c

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/hudson/remoting/SocketChannelStream.java http://jenkins-ci.org/commit/remoting/c90fc463923cc7f4a8907a0b352204f3d561cc55 Log: [FIXED JENKINS-23248] Seeing strange "Transport endpoint is not connected" exception s.shutdownInput() fails with the following exception, even though s.isInputShutdown() is reporting false: java.net.SocketException: Transport endpoint is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:667) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:378) at hudson.remoting.SocketChannelStream$1.close(SocketChannelStream.java:39) at sun.nio.ch.ChannelInputStream.close(ChannelInputStream.java:113) at javax.crypto.CipherInputStream.close(CipherInputStream.java:296) at java.io.BufferedInputStream.close(BufferedInputStream.java:468) at hudson.remoting.FlightRecorderInputStream.close(FlightRecorderInputStream.java:112) at hudson.remoting.ChunkedInputStream.close(ChunkedInputStream.java:102) at hudson.remoting.ChunkedCommandTransport.closeRead(ChunkedCommandTransport.java:50) at hudson.remoting.Channel.terminate(Channel.java:795) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:951) at hudson.remoting.Channel$2.handle(Channel.java:475) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) This bug report may be related: http://bugs.java.com/view_bug.do?bug_id=4516760 If we fail to call s.close(), a socket will leak, so swallowing this exception and have the code execute "s.close()" Compare: https://github.com/jenkinsci/remoting/compare/d1905acf329e...c90fc463923c

          Updated title to reflect the real problem.

          Kohsuke Kawaguchi added a comment - Updated title to reflect the real problem.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          pom.xml
          http://jenkins-ci.org/commit/jenkins/5b7d59a71aa01d0c430d98d0b6879c3662a6248b
          Log:
          [FIXED JENKINS-23248]

          Integrated remoting.jar that fixes the problem

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/5b7d59a71aa01d0c430d98d0b6879c3662a6248b Log: [FIXED JENKINS-23248] Integrated remoting.jar that fixes the problem

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3560
          [FIXED JENKINS-23248] (Revision 5b7d59a71aa01d0c430d98d0b6879c3662a6248b)

          Result = SUCCESS
          kohsuke : 5b7d59a71aa01d0c430d98d0b6879c3662a6248b
          Files :

          • changelog.html
          • pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #3560 [FIXED JENKINS-23248] (Revision 5b7d59a71aa01d0c430d98d0b6879c3662a6248b) Result = SUCCESS kohsuke : 5b7d59a71aa01d0c430d98d0b6879c3662a6248b Files : changelog.html pom.xml

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/0ae16264db1b07c011e76aee81442033caf5c72a
          Log:
          [FIXED JENKINS-23248]

          Integrated remoting.jar that fixes the problem

          (cherry picked from commit 5b7d59a71aa01d0c430d98d0b6879c3662a6248b)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: pom.xml http://jenkins-ci.org/commit/jenkins/0ae16264db1b07c011e76aee81442033caf5c72a Log: [FIXED JENKINS-23248] Integrated remoting.jar that fixes the problem (cherry picked from commit 5b7d59a71aa01d0c430d98d0b6879c3662a6248b) Conflicts: changelog.html

          Jayan MN added a comment - - edited

          Is this really fixed?

          I am on  Jenkins ver. 2.32.3.  Tested with script  to run a dummy job --that seems to have leaked handles - one per invocation.

           

           

          JAVA_CMD=java
          CLI_JAR=/jenkins/jenkins-cli.jar
          JOB_NAME=dummyJob
          URL=http://testserver:8080/

           

          for i in {1..1000}
          do
           $JAVA_CMD -jar $CLI_JAR -s $URL build $JOB_NAME 
          done

           

          Jayan MN added a comment - - edited Is this really fixed? I am on  Jenkins ver. 2.32.3.  Tested with script  to run a dummy job --that seems to have leaked handles - one per invocation.     JAVA_CMD=java CLI_JAR=/jenkins/jenkins-cli.jar JOB_NAME=dummyJob URL= http://testserver:8080/   for i in {1..1000 } do  $JAVA_CMD -jar $CLI_JAR -s $URL build $JOB_NAME  done  

          Daniel Beck added a comment - - edited

          jayanmn There's really no point testing this on an outdated version of Jenkins when 2.46.2 contained a major rewrite of the CLI. If you're affected by an issue that looks like this one, start by upgrading to a recent version of Jenkins.

          If the problem persists (given the age of this issue and major changes since), please file a new bug.

          Daniel Beck added a comment - - edited jayanmn There's really no point testing this on an outdated version of Jenkins when 2.46.2 contained a major rewrite of the CLI. If you're affected by an issue that looks like this one, start by upgrading to a recent version of Jenkins. If the problem persists (given the age of this issue and major changes since), please file a new bug.

          Jayan MN added a comment -

          Thanks, great news.

          I posted my case as this was claimed to be addressed long back in 1.563.... I will pick 2.46.2  and test it in next couple of days.. 

          Jayan MN added a comment - Thanks, great news. I posted my case as this was claimed to be addressed long back in 1.563.... I will pick 2.46.2  and test it in next couple of days.. 

          Daniel Beck added a comment -

          jayanmn

          this was claimed to be addressed long back in 1.563

          Right; an indicator you might be seeing a different bug, so please file a new issue if it's still present. It's very confusing to have multiple reports of the "same" issue that have very different causes.

          Daniel Beck added a comment - jayanmn this was claimed to be addressed long back in 1.563 Right; an indicator you might be seeing a different bug, so please file a new issue if it's still present. It's very confusing to have multiple reports of the "same" issue that have very different causes.

          Jayan MN added a comment -

          tested with Jenkins ver. 2.60.2 - I will create a new bug as you suggested.  

          Jayan MN added a comment - tested with  Jenkins ver. 2.60.2  - I will create a new bug as you suggested.  

            kohsuke Kohsuke Kawaguchi
            iwonbigbro Craig Phillips
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: