Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22456

github plugin too many open files unclosed github-polling.log filehandles causes all builds to fail

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • github-plugin

      Github plugin creates and doesn't close thousands of github-polling.log filehandles, causing all builds to fail. Presents as "too many open files" error on attempting to open a file over the max open files limit, e.g.,

      Caused by: java.io.IOException: Too many open files

      or

      Caused by: java.io.FileNotFoundException: [...] (Too many open files)

      Github Plugin Version: 1.8
      (Jenkins Version: 1.55x)

          [JENKINS-22456] github plugin too many open files unclosed github-polling.log filehandles causes all builds to fail

          webb phillips added a comment - - edited

          Please consider merging this fix: https://github.com/jenkinsci/github-plugin/pull/41

          webb phillips added a comment - - edited Please consider merging this fix: https://github.com/jenkinsci/github-plugin/pull/41

          Jesse Glick added a comment -

          Fix was merged.

          Jesse Glick added a comment - Fix was merged.

          Jesse Glick added a comment -

          Sorry, did not read all the comments there.

          Jesse Glick added a comment - Sorry, did not read all the comments there.

          webb phillips added a comment -

          It looks like the fix wasn't a good fix, and the source of the problem is still unknown.

          webb phillips added a comment - It looks like the fix wasn't a good fix, and the source of the problem is still unknown.

          webb phillips added a comment -

          We've worked around the problem by no longer using the github plugin for polling/queueing builds, so if this doesn't affect anyone else, it's no longer critical.

          Might I suggest though that a better implementation of the polling feature would be to do github polling per repo instead of per job? We have hundreds of jobs using the same repo, and at the least, generating many fewer github-polling.log files would cause the unclosed filehandles to build up much more slowly.

          webb phillips added a comment - We've worked around the problem by no longer using the github plugin for polling/queueing builds, so if this doesn't affect anyone else, it's no longer critical. Might I suggest though that a better implementation of the polling feature would be to do github polling per repo instead of per job? We have hundreds of jobs using the same repo, and at the least, generating many fewer github-polling.log files would cause the unclosed filehandles to build up much more slowly.

          Jesse Glick added a comment -

          If you can reproduce the problem, you can use the File Leak Detector plugin to pinpoint the cause.

          Jesse Glick added a comment - If you can reproduce the problem, you can use the File Leak Detector plugin to pinpoint the cause.

          webb phillips added a comment - - edited

          I confirmed (/file-handles) that job-specific github-polling.log files were not being closed. Running garbage-collection (/gc) and then revisiting open file handles (/file-handles) showed that github-polling.log files from e.g., 45 minutes earlier were still open. Not using the github plugin for polling solved the problem well enough for our needs, and I don't plan to reproduce the problem. If you would like to close as "Won't fix" or "Can't reproduce" that's totally fine by me.

          webb phillips added a comment - - edited I confirmed (/file-handles) that job-specific github-polling.log files were not being closed. Running garbage-collection (/gc) and then revisiting open file handles (/file-handles) showed that github-polling.log files from e.g., 45 minutes earlier were still open. Not using the github plugin for polling solved the problem well enough for our needs, and I don't plan to reproduce the problem. If you would like to close as "Won't fix" or "Can't reproduce" that's totally fine by me.

          Jesse Glick added a comment -

          I installed the File Leak Detector plugin and activated it, but could not reproduce. Various accesses to github-polling.log all reliably close the file. However I also did not manage to get a hook added, despite using a valid API token to connect to my own repo:

          java.io.FileNotFoundException: https://api.github.com/repos/jglick/simple-maven-project-with-tests/hooks
          	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
          	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
          	at org.kohsuke.github.Requester.parse(Requester.java:383)
          	at org.kohsuke.github.Requester._to(Requester.java:185)
          	at org.kohsuke.github.Requester.to(Requester.java:160)
          	at org.kohsuke.github.GHRepository.createHook(GHRepository.java:754)
          	at com.cloudbees.jenkins.GitHubPushTrigger.createJenkinsHook(GitHubPushTrigger.java:161)
          

          Jesse Glick added a comment - I installed the File Leak Detector plugin and activated it, but could not reproduce. Various accesses to github-polling.log all reliably close the file. However I also did not manage to get a hook added, despite using a valid API token to connect to my own repo: java.io.FileNotFoundException: https://api.github.com/repos/jglick/simple-maven-project-with-tests/hooks at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) at org.kohsuke.github.Requester.parse(Requester.java:383) at org.kohsuke.github.Requester._to(Requester.java:185) at org.kohsuke.github.Requester.to(Requester.java:160) at org.kohsuke.github.GHRepository.createHook(GHRepository.java:754) at com.cloudbees.jenkins.GitHubPushTrigger.createJenkinsHook(GitHubPushTrigger.java:161)

          Jesse Glick added a comment -

          Possibly a side effect of JENKINS-27041.

          Jesse Glick added a comment - Possibly a side effect of JENKINS-27041 .

            Unassigned Unassigned
            webb webb phillips
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: