Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43199

Credentials Binding plugin causes File Descriptor leak

      Our deployment of Jenkins with this plugin is currently on an NFS share. In testing a job with this plugin, a file descriptor was apparently left open. We found this out when attempting to delete the test job after it ran. (It was a simple echo job to make sure masking worked as expected, nothing else)

      The issue appears to be related to the anonymous sub-class of the LineTransformationOutputStream class returned by the Wrapper class and similar to the bug resolved in JENKINS-28409 for EnvInject. The sub-class needs to clean up it's parent outputstream and the super class on close.

       

      @Override
       public void close() throws IOException {
       super.close();
       out.close();
       }

          [JENKINS-43199] Credentials Binding plugin causes File Descriptor leak

          This may potentially cause the same issue for the pipeline variant, but is not used by us at this time.

          Elliott Butler added a comment - This may potentially cause the same issue for the pipeline variant, but is not used by us at this time.

          Bill Agee added a comment - - edited

          I'm also seeing this - the versions I'm running are:

          • Jenkins 2.54
          • Credentials Plugin version 2.1.13
          • Credentials Binding Plugin version 1.11

          This issue is causing a pretty serious problem on my Jenkins master, since I have several frequently-running jobs which use the credentials plugin. After a few days those jobs cause Jenkins to have several thousand open FDs to build logs:

          $ sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1
          java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

          My pipeline jobs that use the credentials plugin also appear to be affected by this issue.

          For the moment I've downgraded to Jenkins 2.46.1 and the Credentials Binding Plugin version 1.8, which made the problem go away. It would be nice to get back onto newer releases, though.

          Bill Agee added a comment - - edited I'm also seeing this - the versions I'm running are: Jenkins 2.54 Credentials Plugin version 2.1.13 Credentials Binding Plugin version 1.11 This issue is causing a pretty serious problem on my Jenkins master, since I have several frequently-running jobs which use the credentials plugin. After a few days those jobs cause Jenkins to have several thousand open FDs to build logs: $ sudo sysdig -c lsof " 'proc.pid=3531 and thread.tid=3531' " | tail -n 1 java 3531 3531 jenkins 3204 file / var /lib/jenkins/jobs/myjob/builds/886/log My pipeline jobs that use the credentials plugin also appear to be affected by this issue. For the moment I've downgraded to Jenkins 2.46.1 and the Credentials Binding Plugin version 1.8, which made the problem go away. It would be nice to get back onto newer releases, though.

          Ben Waters added a comment -

          Hi!

           

          Any update? We're using Jenkins 2.53 in our production environments and after update noticed the change. Would really appreciate it.

          Ben Waters added a comment - Hi!   Any update? We're using Jenkins 2.53 in our production environments and after update noticed the change. Would really appreciate it.

          Oleg Nenashev added a comment -

          Somebody was playing with statuses, so nobody picked the "In Review" ticket. Reverted the status to Open

          Oleg Nenashev added a comment - Somebody was playing with statuses, so nobody picked the "In Review" ticket. Reverted the status to Open

          Jesse Glick added a comment -

          From inspecting code, I see no way this would affect Pipeline builds. Freestyle builds yes, due in part to what I think is a core bug in Run.execute: listener.closeQuietly() should instead (or additionally) be logger.close().

          Jesse Glick added a comment - From inspecting code, I see no way this would affect Pipeline builds. Freestyle builds yes, due in part to what I think is a core bug in Run.execute : listener.closeQuietly() should instead (or additionally) be logger.close() .

          Jesse Glick added a comment -

          Cannot find JIRA ID for mssbg to assign to.

          Jesse Glick added a comment - Cannot find JIRA ID for mssbg to assign to.

          For what it's worth, this bug seems to have made it into the LTS release of jenkins (2.60.1) and is incredibly disruptive. We're having to currently restart our jenkins nightly-ish as Ben is doing. I just noticed that we'd upgraded our credentials binding plugin to 1.12, so I'm going to downgrade that to 1.11, which was packaged with jenkins, with tonight's bounce and see if that helps any.

          Abhishek Mukherjee added a comment - For what it's worth, this bug seems to have made it into the LTS release of jenkins (2.60.1) and is incredibly disruptive. We're having to currently restart our jenkins nightly-ish as Ben is doing. I just noticed that we'd upgraded our credentials binding plugin to 1.12, so I'm going to downgrade that to 1.11, which was packaged with jenkins, with tonight's bounce and see if that helps any.

          Oleg Nenashev added a comment -

          Jenkins LTS does not bundle plugins. Once the plugin fix is released, it will become available in previous LTS releases

          Oleg Nenashev added a comment - Jenkins LTS does not bundle plugins. Once the plugin fix is released, it will become available in previous LTS releases

          Ah apologies – uninstall is greyed out and I can only downgrade in the UI so i figured that meant that it came with the jenkins LTS. Fully understand that a plugin can be updated independently (as we've already done to 1.12)

          Abhishek Mukherjee added a comment - Ah apologies – uninstall is greyed out and I can only downgrade in the UI so i figured that meant that it came with the jenkins LTS. Fully understand that a plugin can be updated independently (as we've already done to 1.12)

          Jesse Glick added a comment -

          Depending on the location of the fix; I believe this is actually a core bug, in which case the fix will come in a core update.

          Jesse Glick added a comment - Depending on the location of the fix; I believe this is actually a core bug, in which case the fix will come in a core update.

          Abhishek Mukherjee added a comment - - edited

          Would that be the linked -JENKINS-28409-, by any chance? Thanks!

          EDIT: nvm. just noticed the "Component" of that ticket being a plugin as well. please ignore me

          Abhishek Mukherjee added a comment - - edited Would that be the linked - JENKINS-28409 -, by any chance? Thanks! EDIT: nvm. just noticed the "Component" of that ticket being a plugin as well. please ignore me

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/model/Run.java
          http://jenkins-ci.org/commit/jenkins/a0a55d1da6882e8a56d90885e7cd55a22accbfe1
          Log:
          JENKINS-43199 Reliably close build log file (#2954)

          • Try harder to ensure that the logger is closed at the end of the build.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/a0a55d1da6882e8a56d90885e7cd55a22accbfe1 Log: JENKINS-43199 Reliably close build log file (#2954) JENKINS-43199 Reliably close build log file. Try harder to ensure that the logger is closed at the end of the build.

          Oleg Nenashev added a comment -

          It should be solved by https://github.com/jenkinsci/jenkins/pull/2954 in 2.73

          Oleg Nenashev added a comment - It should be solved by https://github.com/jenkinsci/jenkins/pull/2954 in 2.73

          Jesse Glick added a comment -

          Is there an easy way to reproduce the bug? I did not try to write an automated test for it, so I only believe the fix is effective.

          Jesse Glick added a comment - Is there an easy way to reproduce the bug? I did not try to write an automated test for it, so I only believe the fix is effective.

          Jesse Glick added a comment -

          Technically the lack of a close override in this plugin is not addressed—only the symptom, when using the core patch. Reopening for tracking purposes.

          Jesse Glick added a comment - Technically the lack of a close override in this plugin is not addressed—only the symptom, when using the core patch. Reopening for tracking purposes.

          Oleg Nenashev added a comment -

          Yes, it would be great to get the plugin fixed for older cores

          Oleg Nenashev added a comment - Yes, it would be great to get the plugin fixed for older cores

          Bill Agee added a comment -

          In my environment, the repro steps on Ubuntu 14.04 were:

          • Create a 'Secret text' credential with Global scope, in /credentials/store/system/domain/_/
          • Create a freestyle job
          • Enable and select the secret text value in the job UI, with:
            • Use secret text(s) or file(s) -> Bindings -> Secret text -> Variable set to MY_SECRET
          • Add a step in Build -> Execute shell and put a simple 'exit 0' statement or something similar (this step may not be necessary ... not sure)
          • Run the Jenkins job several times, while using sysdig to examine the pid and thread ID for the process handling the job, and see if you see open file handles related to the job begin stacking up as you repeatedly trigger builds:
          • sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1
            java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

          Bill Agee added a comment - In my environment, the repro steps on Ubuntu 14.04 were: Create a 'Secret text' credential with Global scope, in /credentials/store/system/domain/_/ Create a freestyle job Enable and select the secret text value in the job UI, with: Use secret text(s) or file(s) -> Bindings -> Secret text -> Variable set to MY_SECRET Add a step in Build -> Execute shell and put a simple 'exit 0' statement or something similar (this step may not be necessary ... not sure) Run the Jenkins job several times, while using sysdig to examine the pid and thread ID for the process handling the job, and see if you see open file handles related to the job begin stacking up as you repeatedly trigger builds: sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1 java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

            Unassigned Unassigned
            ebutler Elliott Butler
            Votes:
            6 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: