-
Bug
-
Resolution: Fixed
-
Critical
-
Windows
Jenkins 2.17
-
Powered by SuggestiMate
We are currently unable to update plugins in our jenkins installation, because this exception occurs:
java.io.IOException: Failed to rename C:\Program Files (x86)\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Program Files (x86)\Jenkins\plugins\windows-slaves.jpi
at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)
It doesn't seem to affect all plugins, but it also happends for the JUnit plugin.
- is related to
-
JENKINS-37041 If an older version of a detached plugin is already installed it does not get upgraded
-
- Closed
-
- links to
[JENKINS-37332] java.io.IOException: Failed to rename during Plugin Update
This is happening to me as well.
Windows Slaves Plugin
Failure -
java.io.IOException: Failed to rename C:\Jenkins\plugins\junit.jpi.tmp to C:\Jenkins\plugins\junit.jpi
at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)
JUnit Plugin
Failure -
java.io.IOException: Failed to rename C:\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Jenkins\plugins\windows-slaves.jpi
at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)
On our install, it's happening for the plugins mentioned by others, as well as:
LDAP Plugin
Mailer Plugin
Subversion Plug-in
External Monitor Job Type Plugin
Maven Integration plugin
Javadoc Plugin
Matrix Authorization Strategy Plugin
OWASP Markup Formatter Plugin
Matrix Project Plugin
Ant Plugin
PAM Authentication plugin
All listed plugins are formerly bundled plugins.
What circumstances do these issues appear in? Are you trying to update these plugins?
Would be interesting to know if Jenkins has a lock on these plugin JPIs specifically, and not others, at the time the upgrade is attempted.
Is this happening only since version 2.17, or did this issue occur earlier than that? What Jenkins versions were you running when you last tried to upgrade any plugin in Jeff's list?
I update Jenkins (first) and then any plugins every week. On 8/6, I updated to 2.17 and the all plugin updates available - no issues. On 8/11, there was no Jenkins update available and so I tried to update all plugins and these 2 failed. These are the plugins I updated on 8/11 and on 8/6.
8/11:
Email Extension Plugin This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says. 2.47 2.46 Git Parameter Plug-In This plugin allows you to choose between Git tags or sha1 of your SCM repository so Git Plugin installed is required. 0.6.2 0.6.1 GitHub API Plugin This plugin is a library plugin used by other GitHub related plugins to share the same libraries. This plugin does not have any user visible feature by itself. There's no need to install this plugin manually, although you want to keep it up to date. 1.77 1.76 JUnit Plugin Allows JUnit-format test results to be published. 1.18 1.15 Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. 2.11 2.10 Pipeline: Input Step Adds the Pipeline step ‘input’ to wait for human input or approval. 2.1 2.0 Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. 2.5 2.4 Pipeline: Shared Groovy Libraries Global shared library for Pipeline scripts. 2.2 2.1 Windows Slaves Plugin Allows you to connect to Windows machines and start slave agents on them. 1.2 1.1
8/6:
Durable Task Plugin Library offering an extension point for processes which can run outside of Jenkins yet be monitored. 1.12 1.11 Email Extension Plugin This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says. 2.46 2.44 Git plugin This plugin allows use of Git as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official git client. Use exotic installations at your own risk. 2.5.3 2.5.2 GitHub Organization Folder Plugin Pipeline-as-Code support for a whole GitHub organization. Scans all the branches & repositories in GitHub organization and build them via Jenkins pipelines automatically 1.4 1.3 Pipeline: Basic Steps Commonly used steps for Pipelines. 2.1 2.0 Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. 2.10 2.9 Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. 2.4 2.3 Pipeline: Nodes and Processes Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection. 2.4 2.3 Pipeline: REST API Plugin Provides a REST API to access pipeline and pipeline run data 1.7 1.6 Pipeline: Stage View Plugin Provides a swimlane view of the different stages in a pipeline. 1.7 1.6 Pipeline: Step API API for asynchronous build step primitive. 2.3 2.2 Structs Plugin Library plugin for DSL plugins that need concise names for Jenkins extensions 1.3 1.2
I updated from 1.655 to 2.16 when the issue first surfaced . The issue happens when updating plugins from the web UI.
In Process Explorer, there are in-use JPI files matching exactly the list of plugins that are having issues while updating. The plugins that do not have a problem updating do not show their JPI files as in-use (if this information is of any use, I don't know).
The files seem to be locked by Jenkins. I tried moving windows-slaves.jpi away, but it did not work (because it was in use) until I shut down Jenkins.
Looks like Jenkins is locking the file it is trying to update/overwrite
jeffdafoe nitek To clarify, the plugins locked by Jenkins, if they exist on your instance, are exactly the following list:
https://github.com/jenkinsci/jenkins/blob/b81041c292cc1d2849a95d22381c84e0e1ff26af/core/src/main/java/hudson/ClassicPluginStrategy.java#L413...L427
If (and only if) a plugin is on that list, and exists on your instance, its JPI file is locked?
Happend to me with windows-slave and junit, so yes, only plugins from that list.
danielbeck The list you referenced is exactly the same as the list that I see in sysinternals Process Explorer when I look for locked JPI files.under the jenkins process.
Responsible commit through git bisect (on OS X, with lsof, and running using java -jar jenkins.war):
https://github.com/jenkinsci/jenkins/pull/2489/commits/2a450e5ac2ecd83a54f05c83729852073601816e
$ git bisect good 2a450e5ac2ecd83a54f05c83729852073601816e is the first bad commit commit 2a450e5ac2ecd83a54f05c83729852073601816e Author: Stephen Connolly <stephen.alan.connolly@gmail.com> Date: Thu Jul 28 16:09:21 2016 +0100 [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version :040000 040000 fdfb36c7fafa895b74c030135405fe2920cfc96f d0ead1a29141a585923651e2593c826ce0fdd893 M core $ git bisect log git bisect start # bad: [e02419c61a562d991d8a068a34f357b2342747ca] [maven-release-plugin] prepare release jenkins-2.16 git bisect bad e02419c61a562d991d8a068a34f357b2342747ca # good: [ae59dde643367c90ec4427490b841db9570e32ad] [maven-release-plugin] prepare release jenkins-2.15 git bisect good ae59dde643367c90ec4427490b841db9570e32ad # good: [d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185] Merge pull request #2480 from stephenc/jenkins-36871-prep git bisect good d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185 # bad: [65adca296d42c8038bad27eb1a10b40d71a163f0] Noting merge of JENKINS-37041 git bisect bad 65adca296d42c8038bad27eb1a10b40d71a163f0 # bad: [d92fd4cc63e19c60e374e45e045dcdce2d91e951] [JENKINS-37041] Address Code review comments git bisect bad d92fd4cc63e19c60e374e45e045dcdce2d91e951 # good: [e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc] Merge pull request #2485 from stephenc/jenkins-36996 git bisect good e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc # bad: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version git bisect bad 2a450e5ac2ecd83a54f05c83729852073601816e # good: [8e536513e8d7b08cda63201ba14ee6e30289d713] [JENKINS-36996] Noting merge git bisect good 8e536513e8d7b08cda63201ba14ee6e30289d713 # first bad commit: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
Or maybe not. Trying to locate/fix the issue wasn't all that successful.
oleg_nenashev told me he'll look into this tomorrow, so for record keeping, assigning to him for now.
Here is a cause stacktrace for the runaway file descriptor (on 2.16 war). Fixing the issue.
#46 /Users/nenashev/Documents/jenkins/test/jenkins-2.16/jenkins/plugins/mailer.jpi by thread:Loading bundled plugins on Fri Aug 19 13:50:21 CEST 2016 at java.util.zip.ZipFile.<init>(ZipFile.java:146) at java.util.jar.JarFile.<init>(JarFile.java:154) at java.util.jar.JarFile.<init>(JarFile.java:91) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) at java.net.URL.openStream(URL.java:1037) at hudson.PluginManager.parsePluginManifest(PluginManager.java:975) at hudson.PluginManager.getPluginVersion(PluginManager.java:775) at hudson.PluginManager.getPluginVersion(PluginManager.java:768) at hudson.PluginManager.getPluginVersion(PluginManager.java:756) at hudson.PluginManager.loadDetachedPlugins(PluginManager.java:722) at hudson.LocalPluginManager.loadBundledPlugins(LocalPluginManager.java:85) at hudson.PluginManager$1$1.run(PluginManager.java:374) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:1026) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
So the Jenkins code behaves correctly, but there is an issue with JARURLConnection, which leaks handles in the common use-case, because it does not cleanup underlying resources.
Nice reading: http://www.genevski.com/2010/04/javaneturl-and-jarurlconnection-may.html
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/test/java/hudson/PluginManagerTest.java
http://jenkins-ci.org/commit/jenkins/96c97860b0f018094c20283b15f3ddb3bdd9effe
Log:
[FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)
JENKINS-37332- Improve diagnostics of non-closed streams during reading of the manifests in PluginManager
JENKINS-37332- Leakless processing of JarUrlConnection during Manifest parsing
JENKINS-37332- Also implement leak-safe method for retrieving file modification date
JENKINS-37332- Add spotcheck methods for manifest file access + Javadoc
JENKINS-37332- Also test multi-line and empty attributes in the test
Just hit this after upgrading to LTS 2.7.3 on Windows. If my records are correct I was able to update bundled plugins without issue on LTS 2.7.2. Is the workaround simply to downgrade to 2.7.2 until there's a new LTS release with the current fix?
Or is there a sneaky hack to stay on 2.7.3 and update the bundled plugins?
Thanks
brianeray
Are you sure it is caused by this problem? https://github.com/jenkinsci/jenkins/commit/2a450e5ac2ecd83a54f05c83729852073601816e has not been backported to 2.7.3, hence there should not be such problem.
Could you please run your instance with http://file-leak-detector.kohsuke.org/ and report the list of open file handles?
Oops. Logs say it has been actually backported: https://github.com/jenkinsci/jenkins/commit/6ea7883fed497b68eec9f89ec89082edf1b39ce5
Thank you oleg_nenashev for the quick response and making this an LTS candidate. My process explorer revealed the same leaked file descriptors on the bundled plugin JPIs.
My workaround was to downgrade to 2.7.2, upgrade these plugins, then upgrade once again to 2.7.3.
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/test/java/hudson/PluginManagerTest.java
http://jenkins-ci.org/commit/jenkins/37edc1a3b0c5670c24bd06d157f4de91b99f8391
Log:
[FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)
JENKINS-37332- Improve diagnostics of non-closed streams during reading of the manifests in PluginManager
JENKINS-37332- Leakless processing of JarUrlConnection during Manifest parsing
JENKINS-37332- Also implement leak-safe method for retrieving file modification date
JENKINS-37332- Add spotcheck methods for manifest file access + Javadoc
JENKINS-37332- Also test multi-line and empty attributes in the test
(cherry picked from commit 96c97860b0f018094c20283b15f3ddb3bdd9effe)
danielbeck It has been reopened intentionally since the bug is in LTS now
olivergondza I also have this issue on our Jenkins instance (IIRC with the Mailer plugin).
2.7.4 release discussion: https://groups.google.com/forum/#!topic/jenkinsci-dev/zIq6EgUPQBI
Please vote there if you're affected
I can confirm this issue happening for the windows slaves and the junit plug in.