Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16319

Failure to delete old config files during rekeying on Windows

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Jenkins as a Windows service

      Started re-keying Wed Jan 09 12:59:32 EST 2013
      Scanning C:\JenkinsService\com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper.xml
      ERROR: Failed to rewrite C:\JenkinsService\hudson.scm.CVSSCM.xml
      java.io.IOException: Unable to delete C:\JenkinsService\hudson.scm.CVSSCM.xml
      	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
      	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
      	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
      ERROR: Failed to rewrite C:\JenkinsService\users\andrewg\config.xml
      java.io.IOException: Unable to delete C:\JenkinsService\users\andrewg\config.xml
      	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
      	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
      	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
      	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
      Completed re-keying 0 files on Wed Jan 09 12:59:33 EST 2013
      

      Jenkins always fails to edit and delete these files as the Service will have a lock on them. If I stop the service I won't be able to get to Jenkins to rerun the rekey job.

          [JENKINS-16319] Failure to delete old config files during rekeying on Windows

          I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains.

          Have any workaround for this problem?

          Pei-Tang Huang added a comment - I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains. Have any workaround for this problem?

          This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion

          Wolf Wolfswinkel added a comment - This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion

          alexlombardi added a comment -

          I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346.

          I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD.

          I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.

          alexlombardi added a comment - I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346 . I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD. I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.

          Roman Harmata added a comment -

          It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files":
          ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
          java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
          at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
          at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
          at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)

          For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?

          Roman Harmata added a comment - It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files": ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112) at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143) at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182) For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?

          Jesse Glick added a comment -

          @alexlombardi keep discussion about authentication in JENKINS-16273. This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers).

          @romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem.

          I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.

          Jesse Glick added a comment - @alexlombardi keep discussion about authentication in JENKINS-16273 . This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers). @romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem. I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.

          Damien Finck added a comment -

          Hello,

          I have the same problem on my Server and on my personal computer.

          [...]
          ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml
          [...]

          How I can fix this problem ?

          Instance Jenkins of my Server
          OS : Windows 2008r2 64 bits
          Jenkins : 1.498

          Instance Jenkins of my Personnal Computer
          OS : Windows 7 64 bits
          Jenkins : 1.498 and after update 1.499

          Damien Finck added a comment - Hello, I have the same problem on my Server and on my personal computer. [...] ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml [...] How I can fix this problem ? Instance Jenkins of my Server OS : Windows 2008r2 64 bits Jenkins : 1.498 Instance Jenkins of my Personnal Computer OS : Windows 7 64 bits Jenkins : 1.498 and after update 1.499

          Pei-Tang Huang added a comment - - edited

          @jglick I have following plugin installed and enabled:

          Name Version
          AnsiColor 0.3.1
          Audit Trail 1.7
          Build Flow Plugin 0.6
          Checkstyle Plug-in 3.32
          Claim Plugin 1.7
          Compact Columns 1.9
          Configuration Slicing plugin 1.36
          Copy Artifact Plugin 1.25
          Dashboard View 2.4
          Dependency Graph Viewer Plugin 0.10
          Duplicate Code Scanner Plug-in 2.33
          External Monitor Job Type Plugin 1.1
          FindBugs Plug-in 4.45
          Fortify 360 Plugin 3.6
          Git server plugin 1.1
          javadoc 1.0
          Jenkins Artifact Deployer Plug-in 0.26
          Jenkins Cobertura Plugin 1.8
          Jenkins Continuous Integration game 1.19
          Jenkins disk-usage plugin 0.18
          Jenkins Email Extension Plugin 2.25
          Jenkins GIT plugin 1.1.26
          Jenkins Gravatar plugin 1.1
          Jenkins Job Configuration History Plugin 2.0
          Jenkins jQuery plugin 1.7.2-1
          Jenkins jQuery UI plugin 1.0.2
          Jenkins Mailer Plugin 1.4
          Jenkins promoted builds plugin 2.8
          Jenkins Slave SetupPlugin 1.6
          Jenkins SLOCCount Plug-in 1.8
          Jenkins SSH Slaves plugin 0.22
          Jenkins Subversion Plug-in 1.44
          Jenkins Translation Assistance plugin 1.10
          LDAP Plugin 1.2
          Maven 2 Project Plugin 1.498
          pam-auth 1.0
          PMD Plug-in 3.33
          Priority Sorter 1.3
          Radiator View Plugin 1.13
          Redmine Plugin 0.11-SNAPSHOT (private-12/14/2012 15:48-Tang)
          SCM Sync Configuration Plugin 0.0.6.1
          Static Analysis Collector Plug-in 1.34
          Static Analysis Utilities 1.48
          Task Scanner Plug-in 4.35
          thinBackup 1.6.2
          Timestamper 1.5
          Token Macro Plugin 1.5.1
          View Job Filters 1.22
          Warnings Plugin 4.18
          WAS Builder Plugin 1.6.1

          I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.

          Pei-Tang Huang added a comment - - edited @jglick I have following plugin installed and enabled: Name Version AnsiColor 0.3.1 Audit Trail 1.7 Build Flow Plugin 0.6 Checkstyle Plug-in 3.32 Claim Plugin 1.7 Compact Columns 1.9 Configuration Slicing plugin 1.36 Copy Artifact Plugin 1.25 Dashboard View 2.4 Dependency Graph Viewer Plugin 0.10 Duplicate Code Scanner Plug-in 2.33 External Monitor Job Type Plugin 1.1 FindBugs Plug-in 4.45 Fortify 360 Plugin 3.6 Git server plugin 1.1 javadoc 1.0 Jenkins Artifact Deployer Plug-in 0.26 Jenkins Cobertura Plugin 1.8 Jenkins Continuous Integration game 1.19 Jenkins disk-usage plugin 0.18 Jenkins Email Extension Plugin 2.25 Jenkins GIT plugin 1.1.26 Jenkins Gravatar plugin 1.1 Jenkins Job Configuration History Plugin 2.0 Jenkins jQuery plugin 1.7.2-1 Jenkins jQuery UI plugin 1.0.2 Jenkins Mailer Plugin 1.4 Jenkins promoted builds plugin 2.8 Jenkins Slave SetupPlugin 1.6 Jenkins SLOCCount Plug-in 1.8 Jenkins SSH Slaves plugin 0.22 Jenkins Subversion Plug-in 1.44 Jenkins Translation Assistance plugin 1.10 LDAP Plugin 1.2 Maven 2 Project Plugin 1.498 pam-auth 1.0 PMD Plug-in 3.33 Priority Sorter 1.3 Radiator View Plugin 1.13 Redmine Plugin 0.11-SNAPSHOT (private-12/14/2012 15:48-Tang) SCM Sync Configuration Plugin 0.0.6.1 Static Analysis Collector Plug-in 1.34 Static Analysis Utilities 1.48 Task Scanner Plug-in 4.35 thinBackup 1.6.2 Timestamper 1.5 Token Macro Plugin 1.5.1 View Job Filters 1.22 Warnings Plugin 4.18 WAS Builder Plugin 1.6.1 I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.

          Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck

          Pei-Tang Huang added a comment - Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck

          Damien Finck added a comment -

          Hello,

          I have try to create a new proper instance of Jenkins (1.496) on a Windows Server 2008r2 with no plugins.

          I have juste configure the proxy in Jenkins, and so I have do the update to 1.499.

          When I have start the re-key, I have :
          Re-keying log

          Started re-keying Fri Jan 18 09:00:02 CET 2013
          Scanning C:\Users\Administrateur\.jenkins\hudson.model.UpdateCenter.xml
          ERROR: Failed to rewrite C:\Users\Administrateur\.jenkins\proxy.xml
          java.io.IOException: Unable to delete C:\Users\Administrateur\.jenkins\proxy.xml
          at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
          at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
          at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
          at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
          Completed re-keying 0 files on Fri Jan 18 09:00:02 CET 2013

          Damien Finck added a comment - Hello, I have try to create a new proper instance of Jenkins (1.496) on a Windows Server 2008r2 with no plugins. I have juste configure the proxy in Jenkins, and so I have do the update to 1.499. When I have start the re-key, I have : Re-keying log Started re-keying Fri Jan 18 09:00:02 CET 2013 Scanning C:\Users\Administrateur\.jenkins\hudson.model.UpdateCenter.xml ERROR: Failed to rewrite C:\Users\Administrateur\.jenkins\proxy.xml java.io.IOException: Unable to delete C:\Users\Administrateur\.jenkins\proxy.xml at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112) at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143) at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182) Completed re-keying 0 files on Fri Jan 18 09:00:02 CET 2013

          cforce added a comment -

          Here to on windows serever 2008 with jenkins running on tomcat as windows service with own user in winndows tomcat runs from.
          This user has write/read bright on the complete partition jenkins_home is saved. Till today this alls worked without problems.

          Please provide a fix!

          cforce added a comment - Here to on windows serever 2008 with jenkins running on tomcat as windows service with own user in winndows tomcat runs from. This user has write/read bright on the complete partition jenkins_home is saved. Till today this alls worked without problems. Please provide a fix!

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/util/SecretRewriter.java
          http://jenkins-ci.org/commit/jenkins/8b8231108fb5930bab5c7e2f20685c9a9d237749
          Log:
          [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/util/SecretRewriter.java http://jenkins-ci.org/commit/jenkins/8b8231108fb5930bab5c7e2f20685c9a9d237749 Log: [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2205
          [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows. (Revision 8b8231108fb5930bab5c7e2f20685c9a9d237749)

          Result = SUCCESS
          Jesse Glick : 8b8231108fb5930bab5c7e2f20685c9a9d237749
          Files :

          • changelog.html
          • core/src/main/java/hudson/util/SecretRewriter.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2205 [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows. (Revision 8b8231108fb5930bab5c7e2f20685c9a9d237749) Result = SUCCESS Jesse Glick : 8b8231108fb5930bab5c7e2f20685c9a9d237749 Files : changelog.html core/src/main/java/hudson/util/SecretRewriter.java

          cforce added a comment -

          Still not released, is it?

          cforce added a comment - Still not released, is it?

          Damien Finck added a comment -

          @cforce : You can see the changelog : http://jenkins-ci.org/changelog > "Upcomming changes" > "Rekeying operation (from SECURITY-49 fix in 1.498) failed on Windows. (issue 16319)"

          The fix will be in 1.501

          Damien Finck added a comment - @cforce : You can see the changelog : http://jenkins-ci.org/changelog > "Upcomming changes" > "Rekeying operation (from SECURITY-49 fix in 1.498) failed on Windows. (issue 16319)" The fix will be in 1.501

          Hello,

          same problem in LTS Version Jenkins 1.480.2
          please merge fix to LTS Branch.

          kr.
          hc

          Hans-Christian Starzinger added a comment - Hello, same problem in LTS Version Jenkins 1.480.2 please merge fix to LTS Branch. kr. hc

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/hudson/util/SecretRewriter.java
          http://jenkins-ci.org/commit/jenkins/cb6f200e8663101533f08821adc471b4f6b54fe5
          Log:
          [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.(cherry picked from commit 8b8231108fb5930bab5c7e2f20685c9a9d237749)

          Conflicts:
          changelog.html

          Compare: https://github.com/jenkinsci/jenkins/compare/5f7ad6e5feee...cb6f200e8663

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/util/SecretRewriter.java http://jenkins-ci.org/commit/jenkins/cb6f200e8663101533f08821adc471b4f6b54fe5 Log: [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.(cherry picked from commit 8b8231108fb5930bab5c7e2f20685c9a9d237749) Conflicts: changelog.html Compare: https://github.com/jenkinsci/jenkins/compare/5f7ad6e5feee...cb6f200e8663

          David Odren added a comment - - edited

          An upgrade from Jenkins 1.497 to 1.500 produced the same errors about failure to rewrite the config.xml files. I am running Jenkins as a service on Windows Server 2003 R2 Standard Edition SP2.

          David Odren added a comment - - edited An upgrade from Jenkins 1.497 to 1.500 produced the same errors about failure to rewrite the config.xml files. I am running Jenkins as a service on Windows Server 2003 R2 Standard Edition SP2.

          Jesse Glick added a comment -

          @dodren I reproduced the originally reported bug and confirmed the fix. If you are still seeing an issue the you need to provide the stack trace, with current line numbers.

          Jesse Glick added a comment - @dodren I reproduced the originally reported bug and confirmed the fix. If you are still seeing an issue the you need to provide the stack trace, with current line numbers.

          Damien Finck added a comment -

          It works for me on Jenkins 1.500 !

          Windows Server 2008r2
          Jenkins 1.500

          Damien Finck added a comment - It works for me on Jenkins 1.500 ! Windows Server 2008r2 Jenkins 1.500

          cforce added a comment -

          For me too!

          cforce added a comment - For me too!

          Jesse Glick added a comment -

          @dodren I am reclosing; if you can still reproduce please file a fresh bug with a complete rekeying log and link it to this one. You might have hit some other problem needing a separate fix.

          Jesse Glick added a comment - @dodren I am reclosing; if you can still reproduce please file a fresh bug with a complete rekeying log and link it to this one. You might have hit some other problem needing a separate fix.

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            15 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: