Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8244

"Too Many Open files", Git and SSH slave

XMLWordPrintable

      We have a hudson instance that is failing after only a few builds with
      the "Too Many Open file error". Here are the vitals:

      Hudson 1.386
      Git Plugin 1.1.4

      The system is building a Rails test suite using the Rake plugin. Git
      is configured to build all available branches and there are about 12
      right now, but regular activity only occurs on 2. All builds are
      being done on a slave machine using Unix SSH.

      We used the instructions on the wiki related to the "too many open
      files" error, ls -la /proc/PID/fd shows 2500 files and GC does not
      clean them up. It essentially makes the slave useless until we
      disconnect and reconnect to it.

      If we don't have git configured to use all the branches then we don't
      get the problem.

      Here is a full backtrace The Git error at the top can vary

      /usr/bin/git checkout -f 7e42dcecfc5a792e9865d01916cb1e8942jkfc35
      FATAL: Could not checkout 7e42dcecfc5a792e9865d01916cb1e8942jkfc35
      hudson.plugins.git.GitException: Could not checkout 7e42dcecfc5a792e9865d01916cb1e8942jkfc35
      at hudson.plugins.git.GitAPI.checkout(GitAPI.java:450)
      at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:981)
      at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:972)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1899)
      at hudson.remoting.UserRequest.perform(UserRequest.java:114)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:270)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)
      Caused by: hudson.plugins.git.GitException: Error performing /usr/bin/git checkout -f 7e42dcecfc5a792e9865d01916cb1e8942jkfc35
      Cannot run program "/usr/bin/git" (in directory "/home/web/workspace/MyProject"): java.io.IOException: error=24, Too many open files
      at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:379)
      at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:344)
      at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:354)
      at hudson.plugins.git.GitAPI.checkout(GitAPI.java:448)
      ... 12 more
      Caused by: java.io.IOException: Cannot run program "/usr/bin/git" (in directory "/home/web/workspace/MyProject"): java.io.IOException: error=24, Too many open files
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
      at hudson.Proc$LocalProc.<init>(Proc.java:192)
      at hudson.Proc$LocalProc.<init>(Proc.java:164)
      at hudson.Launcher$LocalLauncher.launch(Launcher.java:638)
      at hudson.Launcher$ProcStarter.start(Launcher.java:273)
      at hudson.Launcher$ProcStarter.join(Launcher.java:280)
      at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:368)
      ... 15 more
      Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
      at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
      at java.lang.ProcessImpl.start(ProcessImpl.java:81)
      at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
      ... 21 more

            abayer Andrew Bayer
            pallan pallan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: