Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16319

Failure to delete old config files during rekeying on Windows

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Jenkins as a Windows service

      Started re-keying Wed Jan 09 12:59:32 EST 2013
      Scanning C:\JenkinsService\com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper.xml
      ERROR: Failed to rewrite C:\JenkinsService\hudson.scm.CVSSCM.xml
      java.io.IOException: Unable to delete C:\JenkinsService\hudson.scm.CVSSCM.xml
      	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
      	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
      	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
      ERROR: Failed to rewrite C:\JenkinsService\users\andrewg\config.xml
      java.io.IOException: Unable to delete C:\JenkinsService\users\andrewg\config.xml
      	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
      	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
      	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
      Completed re-keying 0 files on Wed Jan 09 12:59:33 EST 2013
      

      Jenkins always fails to edit and delete these files as the Service will have a lock on them. If I stop the service I won't be able to get to Jenkins to rerun the rekey job.

          [JENKINS-16319] Failure to delete old config files during rekeying on Windows

          Jesse Glick created issue -
          Jesse Glick made changes -
          Link New: This issue is blocking SECURITY-49 [ SECURITY-49 ]

          I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains.

          Have any workaround for this problem?

          Pei-Tang Huang added a comment - I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains. Have any workaround for this problem?

          This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion

          Wolf Wolfswinkel added a comment - This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion

          alexlombardi added a comment -

          I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346.

          I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD.

          I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.

          alexlombardi added a comment - I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346 . I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD. I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.

          Roman Harmata added a comment -

          It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files":
          ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
          java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
          at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
          at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
          at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)

          For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?

          Roman Harmata added a comment - It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files": ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112) at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143) at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182) For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?

          Jesse Glick added a comment -

          @alexlombardi keep discussion about authentication in JENKINS-16273. This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers).

          @romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem.

          I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.

          Jesse Glick added a comment - @alexlombardi keep discussion about authentication in JENKINS-16273 . This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers). @romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem. I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.

          Damien Finck added a comment -

          Hello,

          I have the same problem on my Server and on my personal computer.

          [...]
          ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml
          [...]

          How I can fix this problem ?

          Instance Jenkins of my Server
          OS : Windows 2008r2 64 bits
          Jenkins : 1.498

          Instance Jenkins of my Personnal Computer
          OS : Windows 7 64 bits
          Jenkins : 1.498 and after update 1.499

          Damien Finck added a comment - Hello, I have the same problem on my Server and on my personal computer. [...] ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml [...] How I can fix this problem ? Instance Jenkins of my Server OS : Windows 2008r2 64 bits Jenkins : 1.498 Instance Jenkins of my Personnal Computer OS : Windows 7 64 bits Jenkins : 1.498 and after update 1.499

          Pei-Tang Huang added a comment - - edited

          @jglick I have following plugin installed and enabled:

          Name Version
          AnsiColor 0.3.1
          Audit Trail 1.7
          Build Flow Plugin 0.6
          Checkstyle Plug-in 3.32
          Claim Plugin 1.7
          Compact Columns 1.9
          Configuration Slicing plugin 1.36
          Copy Artifact Plugin 1.25
          Dashboard View 2.4
          Dependency Graph Viewer Plugin 0.10
          Duplicate Code Scanner Plug-in 2.33
          External Monitor Job Type Plugin 1.1
          FindBugs Plug-in 4.45
          Fortify 360 Plugin 3.6
          Git server plugin 1.1
          javadoc 1.0
          Jenkins Artifact Deployer Plug-in 0.26
          Jenkins Cobertura Plugin 1.8
          Jenkins Continuous Integration game 1.19
          Jenkins disk-usage plugin 0.18
          Jenkins Email Extension Plugin 2.25
          Jenkins GIT plugin 1.1.26
          Jenkins Gravatar plugin 1.1
          Jenkins Job Configuration History Plugin 2.0
          Jenkins jQuery plugin 1.7.2-1
          Jenkins jQuery UI plugin 1.0.2
          Jenkins Mailer Plugin 1.4
          Jenkins promoted builds plugin 2.8
          Jenkins Slave SetupPlugin 1.6
          Jenkins SLOCCount Plug-in 1.8
          Jenkins SSH Slaves plugin 0.22
          Jenkins Subversion Plug-in 1.44
          Jenkins Translation Assistance plugin 1.10
          LDAP Plugin 1.2
          Maven 2 Project Plugin 1.498
          pam-auth 1.0
          PMD Plug-in 3.33
          Priority Sorter 1.3
          Radiator View Plugin 1.13
          Redmine Plugin 0.11-SNAPSHOT (private-12/14/2012 15:48-Tang)
          SCM Sync Configuration Plugin 0.0.6.1
          Static Analysis Collector Plug-in 1.34
          Static Analysis Utilities 1.48
          Task Scanner Plug-in 4.35
          thinBackup 1.6.2
          Timestamper 1.5
          Token Macro Plugin 1.5.1
          View Job Filters 1.22
          Warnings Plugin 4.18
          WAS Builder Plugin 1.6.1

          I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.

          Pei-Tang Huang added a comment - - edited @jglick I have following plugin installed and enabled: Name Version AnsiColor 0.3.1 Audit Trail 1.7 Build Flow Plugin 0.6 Checkstyle Plug-in 3.32 Claim Plugin 1.7 Compact Columns 1.9 Configuration Slicing plugin 1.36 Copy Artifact Plugin 1.25 Dashboard View 2.4 Dependency Graph Viewer Plugin 0.10 Duplicate Code Scanner Plug-in 2.33 External Monitor Job Type Plugin 1.1 FindBugs Plug-in 4.45 Fortify 360 Plugin 3.6 Git server plugin 1.1 javadoc 1.0 Jenkins Artifact Deployer Plug-in 0.26 Jenkins Cobertura Plugin 1.8 Jenkins Continuous Integration game 1.19 Jenkins disk-usage plugin 0.18 Jenkins Email Extension Plugin 2.25 Jenkins GIT plugin 1.1.26 Jenkins Gravatar plugin 1.1 Jenkins Job Configuration History Plugin 2.0 Jenkins jQuery plugin 1.7.2-1 Jenkins jQuery UI plugin 1.0.2 Jenkins Mailer Plugin 1.4 Jenkins promoted builds plugin 2.8 Jenkins Slave SetupPlugin 1.6 Jenkins SLOCCount Plug-in 1.8 Jenkins SSH Slaves plugin 0.22 Jenkins Subversion Plug-in 1.44 Jenkins Translation Assistance plugin 1.10 LDAP Plugin 1.2 Maven 2 Project Plugin 1.498 pam-auth 1.0 PMD Plug-in 3.33 Priority Sorter 1.3 Radiator View Plugin 1.13 Redmine Plugin 0.11-SNAPSHOT (private-12/14/2012 15:48-Tang) SCM Sync Configuration Plugin 0.0.6.1 Static Analysis Collector Plug-in 1.34 Static Analysis Utilities 1.48 Task Scanner Plug-in 4.35 thinBackup 1.6.2 Timestamper 1.5 Token Macro Plugin 1.5.1 View Job Filters 1.22 Warnings Plugin 4.18 WAS Builder Plugin 1.6.1 I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.

          Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck

          Pei-Tang Huang added a comment - Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            15 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: