I can confirm this issue happening for the windows slaves and the junit plug in.

This is happening to me as well.

Windows Slaves Plugin
Failure -
java.io.IOException: Failed to rename C:\Jenkins\plugins\junit.jpi.tmp to C:\Jenkins\plugins\junit.jpi
at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)

JUnit Plugin
Failure -
java.io.IOException: Failed to rename C:\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Jenkins\plugins\windows-slaves.jpi
at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Unknown Source)

On our install, it's happening for the plugins mentioned by others, as well as:
LDAP Plugin
Mailer Plugin
Subversion Plug-in
External Monitor Job Type Plugin
Maven Integration plugin
Javadoc Plugin
Matrix Authorization Strategy Plugin
OWASP Markup Formatter Plugin
Matrix Project Plugin
Ant Plugin
PAM Authentication plugin

All listed plugins are formerly bundled plugins.

What circumstances do these issues appear in? Are you trying to update these plugins?

Would be interesting to know if Jenkins has a lock on these plugin JPIs specifically, and not others, at the time the upgrade is attempted.

Is this happening only since version 2.17, or did this issue occur earlier than that? What Jenkins versions were you running when you last tried to upgrade any plugin in Jeff's list?

I update Jenkins (first) and then any plugins every week. On 8/6, I updated to 2.17 and the all plugin updates available - no issues. On 8/11, there was no Jenkins update available and so I tried to update all plugins and these 2 failed. These are the plugins I updated on 8/11 and on 8/6.

8/11:

	Email Extension Plugin
This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says.	2.47	2.46
	Git Parameter Plug-In
This plugin allows you to choose between Git tags or sha1 of your SCM repository so Git Plugin installed is required.	0.6.2	0.6.1
	GitHub API Plugin
This plugin is a library plugin used by other GitHub related plugins to share the same libraries. This plugin does not have any user visible feature by itself. There's no need to install this plugin manually, although you want to keep it up to date.	1.77	1.76
	JUnit Plugin
Allows JUnit-format test results to be published.	1.18	1.15
	Pipeline: Groovy
Pipeline execution engine based on continuation passing style transformation of Groovy scripts.	2.11	2.10
	Pipeline: Input Step
Adds the Pipeline step ‘input’ to wait for human input or approval.	2.1	2.0
	Pipeline: Job
Defines a new job type for pipelines and provides their generic user interface.	2.5	2.4
	Pipeline: Shared Groovy Libraries
Global shared library for Pipeline scripts.	2.2	2.1
	Windows Slaves Plugin
Allows you to connect to Windows machines and start slave agents on them.	1.2	1.1

8/6:

	Durable Task Plugin
Library offering an extension point for processes which can run outside of Jenkins yet be monitored.	1.12	1.11
	Email Extension Plugin
This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says.	2.46	2.44
	Git plugin
This plugin allows use of Git as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official git client. Use exotic installations at your own risk.
2.5.3	2.5.2
	GitHub Organization Folder Plugin
Pipeline-as-Code support for a whole GitHub organization. Scans all the branches & repositories in GitHub organization and build them via Jenkins pipelines automatically
1.4	1.3
	Pipeline: Basic Steps
Commonly used steps for Pipelines.	2.1	2.0
	Pipeline: Groovy
Pipeline execution engine based on continuation passing style transformation of Groovy scripts.	2.10	2.9
	Pipeline: Job
Defines a new job type for pipelines and provides their generic user interface.	2.4	2.3
	Pipeline: Nodes and Processes
Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection.	2.4	2.3
	Pipeline: REST API Plugin
Provides a REST API to access pipeline and pipeline run data	1.7	1.6
	Pipeline: Stage View Plugin
Provides a swimlane view of the different stages in a pipeline.	1.7	1.6
	Pipeline: Step API
API for asynchronous build step primitive.	2.3	2.2
	Structs Plugin
Library plugin for DSL plugins that need concise names for Jenkins extensions	1.3	1.2

I updated from 1.655 to 2.16 when the issue first surfaced . The issue happens when updating plugins from the web UI.

In Process Explorer, there are in-use JPI files matching exactly the list of plugins that are having issues while updating. The plugins that do not have a problem updating do not show their JPI files as in-use (if this information is of any use, I don't know).

The files seem to be locked by Jenkins. I tried moving windows-slaves.jpi away, but it did not work (because it was in use) until I shut down Jenkins.
Looks like Jenkins is locking the file it is trying to update/overwrite

jeffdafoe nitek To clarify, the plugins locked by Jenkins, if they exist on your instance, are exactly the following list:
https://github.com/jenkinsci/jenkins/blob/b81041c292cc1d2849a95d22381c84e0e1ff26af/core/src/main/java/hudson/ClassicPluginStrategy.java#L413...L427

If (and only if) a plugin is on that list, and exists on your instance, its JPI file is locked?

Happend to me with windows-slave and junit, so yes, only plugins from that list.

danielbeck The list you referenced is exactly the same as the list that I see in sysinternals Process Explorer when I look for locked JPI files.under the jenkins process.

Responsible commit through git bisect (on OS X, with lsof, and running using java -jar jenkins.war):
https://github.com/jenkinsci/jenkins/pull/2489/commits/2a450e5ac2ecd83a54f05c83729852073601816e

$ git bisect good
2a450e5ac2ecd83a54f05c83729852073601816e is the first bad commit
commit 2a450e5ac2ecd83a54f05c83729852073601816e
Author: Stephen Connolly <stephen.alan.connolly@gmail.com>
Date:   Thu Jul 28 16:09:21 2016 +0100

    [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version

:040000 040000 fdfb36c7fafa895b74c030135405fe2920cfc96f d0ead1a29141a585923651e2593c826ce0fdd893 M      core

$ git bisect log
git bisect start
# bad: [e02419c61a562d991d8a068a34f357b2342747ca] [maven-release-plugin] prepare release jenkins-2.16
git bisect bad e02419c61a562d991d8a068a34f357b2342747ca
# good: [ae59dde643367c90ec4427490b841db9570e32ad] [maven-release-plugin] prepare release jenkins-2.15
git bisect good ae59dde643367c90ec4427490b841db9570e32ad
# good: [d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185] Merge pull request #2480 from stephenc/jenkins-36871-prep
git bisect good d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185
# bad: [65adca296d42c8038bad27eb1a10b40d71a163f0] Noting merge of JENKINS-37041
git bisect bad 65adca296d42c8038bad27eb1a10b40d71a163f0
# bad: [d92fd4cc63e19c60e374e45e045dcdce2d91e951] [JENKINS-37041] Address Code review comments
git bisect bad d92fd4cc63e19c60e374e45e045dcdce2d91e951
# good: [e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc] Merge pull request #2485 from stephenc/jenkins-36996
git bisect good e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc
# bad: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
git bisect bad 2a450e5ac2ecd83a54f05c83729852073601816e
# good: [8e536513e8d7b08cda63201ba14ee6e30289d713] [JENKINS-36996] Noting merge
git bisect good 8e536513e8d7b08cda63201ba14ee6e30289d713
# first bad commit: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version

Or maybe not. Trying to locate/fix the issue wasn't all that successful.

oleg_nenashev told me he'll look into this tomorrow, so for record keeping, assigning to him for now.

Here is a cause stacktrace for the runaway file descriptor (on 2.16 war). Fixing the issue.

#46 /Users/nenashev/Documents/jenkins/test/jenkins-2.16/jenkins/plugins/mailer.jpi by thread:Loading bundled plugins on Fri Aug 19 13:50:21 CEST 2016
	at java.util.zip.ZipFile.<init>(ZipFile.java:146)
	at java.util.jar.JarFile.<init>(JarFile.java:154)
	at java.util.jar.JarFile.<init>(JarFile.java:91)
	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
	at java.net.URL.openStream(URL.java:1037)
	at hudson.PluginManager.parsePluginManifest(PluginManager.java:975)
	at hudson.PluginManager.getPluginVersion(PluginManager.java:775)
	at hudson.PluginManager.getPluginVersion(PluginManager.java:768)
	at hudson.PluginManager.getPluginVersion(PluginManager.java:756)
	at hudson.PluginManager.loadDetachedPlugins(PluginManager.java:722)
	at hudson.LocalPluginManager.loadBundledPlugins(LocalPluginManager.java:85)
	at hudson.PluginManager$1$1.run(PluginManager.java:374)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
	at jenkins.model.Jenkins$7.runTask(Jenkins.java:1026)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

So the Jenkins code behaves correctly, but there is an issue with JARURLConnection, which leaks handles in the common use-case, because it does not cleanup underlying resources.

Nice reading: http://www.genevski.com/2010/04/javaneturl-and-jarurlconnection-may.html

Pull request: https://github.com/jenkinsci/jenkins/pull/2516

Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/test/java/hudson/PluginManagerTest.java
http://jenkins-ci.org/commit/jenkins/96c97860b0f018094c20283b15f3ddb3bdd9effe
Log:
[FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)

• JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager

• JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing

• JENKINS-37332 - Also implement leak-safe method for retrieving file modification date

• JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc

• JENKINS-37332 - Also test multi-line and empty attributes in the test

The fix should be released in 2.19

Just hit this after upgrading to LTS 2.7.3 on Windows. If my records are correct I was able to update bundled plugins without issue on LTS 2.7.2. Is the workaround simply to downgrade to 2.7.2 until there's a new LTS release with the current fix?

Or is there a sneaky hack to stay on 2.7.3 and update the bundled plugins?

Thanks

The fix should have been integraed into 2.7.3 Checking

Well, it was not
CC danielbeck olivergondza

brianeray
Are you sure it is caused by this problem? https://github.com/jenkinsci/jenkins/commit/2a450e5ac2ecd83a54f05c83729852073601816e has not been backported to 2.7.3, hence there should not be such problem.

Could you please run your instance with http://file-leak-detector.kohsuke.org/ and report the list of open file handles?

Oops. Logs say it has been actually backported: https://github.com/jenkinsci/jenkins/commit/6ea7883fed497b68eec9f89ec89082edf1b39ce5

Thank you oleg_nenashev for the quick response and making this an LTS candidate. My process explorer revealed the same leaked file descriptors on the bundled plugin JPIs.

My workaround was to downgrade to 2.7.2, upgrade these plugins, then upgrade once again to 2.7.3.

Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/PluginManager.java
core/src/main/java/hudson/Util.java
core/src/test/java/hudson/PluginManagerTest.java
http://jenkins-ci.org/commit/jenkins/37edc1a3b0c5670c24bd06d157f4de91b99f8391
Log:
[FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)

• JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager

• JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing

• JENKINS-37332 - Also implement leak-safe method for retrieving file modification date

• JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc

• JENKINS-37332 - Also test multi-line and empty attributes in the test

(cherry picked from commit 96c97860b0f018094c20283b15f3ddb3bdd9effe)

danielbeck It has been reopened intentionally since the bug is in LTS now

olivergondza I also have this issue on our Jenkins instance (IIRC with the Mailer plugin).

2.7.4 release discussion: https://groups.google.com/forum/#!topic/jenkinsci-dev/zIq6EgUPQBI
Please vote there if you're affected

Fixed on 2.7 LTS line in 2.7.4.