Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65738

File handle leak in timestamper plugin

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: timestamper-plugin
    • Labels:
      None
    • Environment:
      Jenkins LTS 2.263.4
      timestamper-plugin 1.13
      OpenJDK 1.8.0_282
      CentOS 7.9.2009
    • Similar Issues:

      Description

      Since updating timestamper-plugin to version 1.13, Jenkins throws IOExceptions for "Too many open files" after some hours and Jenkins gets unresponsive.

      Checked with Linux's "lsof -u jenkins" command we see a lot of open files with pattern

      /var/lib/jenkins/jobs/*/builds/*/timestamper/timestamps

      The number of open files of this pattern increases over time until the limit is reached.

      A Jenkins restart frees the file handles and makes Jenkins responsive again - until open file limits are reached again.

      A downgrade of timestamper-plugin to version 1.12 solved the issue for us.

       

        Attachments

          Activity

          Hide
          basil Basil Crow added a comment -

          Thanks for reporting this, Michael Meissner. I did indeed change the code in TimestampsReader and TimestampsWriter from new FileInputStream and new FileOutputStream to Files.newInputStream and Files.newOutputStream in 1.13. However, I just audited the new code and I do not see any obvious file handle leaks. And interactive testing of 1.13 on 2.277.4 LTS while watching the open file table did not reveal any problems either: I was able to create and run multiple timestamped Freestyle jobs concurrently, viewing the console logs in my browser, and at the conclusion of it all there were no timestamps files left open. Timestamper 1.13 was also released 3 weeks ago, and I find it surprising that no other users have complained besides you. If there was truly a leak that affected all users, I would expect to have heard from more people by now.

          I think whatever is going on must be specific to your Jenkins installation. It's actually normal for the timestamps files to be open while users are viewing console logs, but those files should be closed once the thread handling the HTTP request finishes its processing. I confirmed this interactively with 1.13 and 2.277.4 LTS. I wonder if this might be a red herring and your real problem is with some other open files. Or maybe you are having a problem with some other plugin.

          Since I can't reproduce this based on the information you've given me, I am going to need a little help from your side to get to the bottom of this. Can you run Jenkins with the File Leak Detector agent? Then when you hit a "Too many open files" error, the agent will dump a table to standard error with the list of open files and the stack trace of the thread that opened the file. Send me that complete table and I can possibly help.

          Show
          basil Basil Crow added a comment - Thanks for reporting this, Michael Meissner . I did indeed change the code in TimestampsReader and TimestampsWriter from new FileInputStream and new FileOutputStream to Files.newInputStream and Files.newOutputStream in 1.13. However, I just audited the new code and I do not see any obvious file handle leaks. And interactive testing of 1.13 on 2.277.4 LTS while watching the open file table did not reveal any problems either: I was able to create and run multiple timestamped Freestyle jobs concurrently, viewing the console logs in my browser, and at the conclusion of it all there were no timestamps files left open. Timestamper 1.13 was also released 3 weeks ago, and I find it surprising that no other users have complained besides you. If there was truly a leak that affected all users, I would expect to have heard from more people by now. I think whatever is going on must be specific to your Jenkins installation. It's actually normal for the timestamps files to be open while users are viewing console logs, but those files should be closed once the thread handling the HTTP request finishes its processing. I confirmed this interactively with 1.13 and 2.277.4 LTS. I wonder if this might be a red herring and your real problem is with some other open files. Or maybe you are having a problem with some other plugin. Since I can't reproduce this based on the information you've given me, I am going to need a little help from your side to get to the bottom of this. Can you run Jenkins with the File Leak Detector agent? Then when you hit a "Too many open files" error, the agent will dump a table to standard error with the list of open files and the stack trace of the thread that opened the file. Send me that complete table and I can possibly help.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            lenslord Michael Meissner
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: