Seems that JAR caching also takes DLL files on Windows.
      Thanks to Windows architecture (), it leads to access conflicts, because DDLs may be locked by other process.

      The issue is being heavily reproduced during the process. After several hours, any process termination fails with the FileNotFoundException exception (see the log below).

      Probably, the high frequency of the issue is somehow related to "Cygwin Process Killer" plugin. On my installation it calls additional remote call on the node before any process termination (it is an interim workaround for JENKINS-19156 and JENKINS-20289 in the custom 1.509.4 core build).

      Dec 07, 2013 7:02:28 AM org.jvnet.winp.Native load
      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\XXX\.jenkins\cache\jars\01\winp.x64.dll (The process cannot access the file because it is being used by another process)
      at java.io.FileOutputStream.open(Native Method)
      at java.io.FileOutputStream.<init>(Unknown Source)
      at java.io.FileOutputStream.<init>(Unknown Source)
      at org.jvnet.winp.Native.load(Native.java:81)
      at org.jvnet.winp.Native.<clinit>(Native.java:52)
      at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
      at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:470)
      at hudson.util.ProcessTree.get(ProcessTree.java:335)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

      Dec 07, 2013 7:02:35 AM hudson.util.ProcessTree get
      WARNING: Failed to load winp. Reverting to the default
      java.lang.UnsatisfiedLinkError: Native Library C:\Users\XXX\.jenkins\cache\jars\01\winp.x64.dll already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary1(Unknown Source)
      at java.lang.ClassLoader.loadLibrary0(Unknown Source)
      at java.lang.ClassLoader.loadLibrary(Unknown Source)
      at java.lang.Runtime.load0(Unknown Source)
      at java.lang.System.load(Unknown Source)
      at org.jvnet.winp.Native.loadDll(Native.java:158)
      at org.jvnet.winp.Native.load(Native.java:90)
      at org.jvnet.winp.Native.<clinit>(Native.java:52)
      at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
      at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:470)
      at hudson.util.ProcessTree.get(ProcessTree.java:335)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

          [JENKINS-20913] [jar caching] - winp.x64.dll access conflicts

          This bug is not even marked as fixed yet, so it's way too early to predict LTS backporting plan. We'd first have to fix this in the trunk.

          Knowing whether the library is already loaded or not doesn't really help either, because if it's loaded through another classloader, we won't be able to reuse currently loaded copy.

          Kohsuke Kawaguchi added a comment - This bug is not even marked as fixed yet, so it's way too early to predict LTS backporting plan. We'd first have to fix this in the trunk. Knowing whether the library is already loaded or not doesn't really help either, because if it's loaded through another classloader, we won't be able to reuse currently loaded copy.

          I'm now marking this issue fixed with winp 1.19.

          The other part of the issue (of not reusing the same JVM for multiple reconnections to the master) will be tracked in JENKINS-19055,

          Kohsuke Kawaguchi added a comment - I'm now marking this issue fixed with winp 1.19. The other part of the issue (of not reusing the same JVM for multiple reconnections to the master) will be tracked in JENKINS-19055 ,

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/4d969004c121bcef07189dfe89fe3347646e741a
          Log:
          JENKINS-20913 Use winp 1.19

          Updated to a newer version

          (cherry picked from commit 6b350af44dee8e1e6cad4cce67941b3ebf88ac52)

          Conflicts:
          core/pom.xml

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/4d969004c121bcef07189dfe89fe3347646e741a Log: JENKINS-20913 Use winp 1.19 Updated to a newer version (cherry picked from commit 6b350af44dee8e1e6cad4cce67941b3ebf88ac52) Conflicts: core/pom.xml

          Arie Belenky added a comment -

          Hi,
          This issue started to reproduce during the last week on our AWS Windows Server 2012 R2 base machines.
          Jenkins ver. 1.640
          Traceback:
          SEVERE: I/O error in channel channel
          java.net.SocketException: Connection reset
          at java.net.SocketInputStream.read(Unknown Source)
          at java.net.SocketInputStream.read(Unknown Source)
          at java.io.BufferedInputStream.fill(Unknown Source)
          at java.io.BufferedInputStream.read(Unknown Source)
          at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
          at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
          at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
          at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

          Dec 31, 2015 5:17:38 PM hudson.remoting.Request$2 run
          SEVERE: Failed to send back a reply
          java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(Unknown Source)
          at java.net.SocketOutputStream.write(Unknown Source)
          at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
          at java.io.BufferedOutputStream.write(Unknown Source)
          at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:94)
          at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:89)
          at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:58)
          at java.io.OutputStream.write(Unknown Source)
          at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
          at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:45)
          at hudson.remoting.Channel.send(Channel.java:582)
          at hudson.remoting.Request$2.run(Request.java:340)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:62)
          at java.lang.Thread.run(Unknown Source)

          Dec 31, 2015 5:17:38 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Terminated
          Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among https://jenkins.interjunk.com/
          Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to jenkins.interjunk.com:15400
          Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Trying protocol: JNLP2-connect
          Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected
          Dec 31, 2015 5:19:00 PM hudson.util.ProcessTree get
          WARNING: Failed to load winp. Reverting to the default
          java.lang.UnsatisfiedLinkError: Native Library C:\Users\automation\.jenkins\cache\jars\4A\winp.x64.22D9AB310A3FA2D96B6E03A836A47724.dll
          already loaded in another classloader
          at java.lang.ClassLoader.loadLibrary1(Unknown Source)
          at java.lang.ClassLoader.loadLibrary0(Unknown Source)
          at java.lang.ClassLoader.loadLibrary(Unknown Source)
          at java.lang.Runtime.load0(Unknown Source)
          at java.lang.System.load(Unknown Source)
          at org.jvnet.winp.Native.loadDll(Native.java:190)
          at org.jvnet.winp.Native.load(Native.java:122)
          at org.jvnet.winp.Native.<clinit>(Native.java:56)
          at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212)
          at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:494)
          at hudson.util.ProcessTree.get(ProcessTree.java:345)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:965)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:956)
          at hudson.remoting.UserRequest.perform(UserRequest.java:120)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:326)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:62)
          at java.lang.Thread.run(Unknown Source)

          Arie Belenky added a comment - Hi, This issue started to reproduce during the last week on our AWS Windows Server 2012 R2 base machines. Jenkins ver. 1.640 Traceback: SEVERE: I/O error in channel channel java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Dec 31, 2015 5:17:38 PM hudson.remoting.Request$2 run SEVERE: Failed to send back a reply java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.flushBuffer(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:94) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:89) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:58) at java.io.OutputStream.write(Unknown Source) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:45) at hudson.remoting.Channel.send(Channel.java:582) at hudson.remoting.Request$2.run(Request.java:340) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:62) at java.lang.Thread.run(Unknown Source) Dec 31, 2015 5:17:38 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among https://jenkins.interjunk.com/ Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.interjunk.com:15400 Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Dec 31, 2015 5:19:00 PM hudson.util.ProcessTree get WARNING: Failed to load winp. Reverting to the default java.lang.UnsatisfiedLinkError: Native Library C:\Users\automation\.jenkins\cache\jars\4A\winp.x64.22D9AB310A3FA2D96B6E03A836A47724.dll already loaded in another classloader at java.lang.ClassLoader.loadLibrary1(Unknown Source) at java.lang.ClassLoader.loadLibrary0(Unknown Source) at java.lang.ClassLoader.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at org.jvnet.winp.Native.loadDll(Native.java:190) at org.jvnet.winp.Native.load(Native.java:122) at org.jvnet.winp.Native.<clinit>(Native.java:56) at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212) at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:494) at hudson.util.ProcessTree.get(ProcessTree.java:345) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:965) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:956) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:62) at java.lang.Thread.run(Unknown Source)

          Hi, this issue is reproduced on our servers.

          Jenkins ver. 1.625.1 :
          Master on CentOS 6
          Node in Windows 7, slave running as service, working !! (ie. NO java.lang.UnsatisfiedLinkError raised with winp dll)
          Node in Windows 7, slave running via CLA, NOT working (ie. java.lang.UnsatisfiedLinkError raised with winp dll)
          Node in Windows 8.1, not working in both cases ( as service, or via CLI)

          Pierre-Henri Cazes added a comment - Hi, this issue is reproduced on our servers. Jenkins ver. 1.625.1 : Master on CentOS 6 Node in Windows 7, slave running as service, working !! (ie. NO java.lang.UnsatisfiedLinkError raised with winp dll) Node in Windows 7, slave running via CLA, NOT working (ie. java.lang.UnsatisfiedLinkError raised with winp dll) Node in Windows 8.1, not working in both cases ( as service, or via CLI)

          ci_jenkinsci_org, this bug seems to be caused by winp library issue https://github.com/kohsuke/winp/issues/26. Could you please promote a new release of fixed winp and update Jenkins to it in next LTS?

          Grigory Merkin added a comment - ci_jenkinsci_org , this bug seems to be caused by winp library issue https://github.com/kohsuke/winp/issues/26 . Could you please promote a new release of fixed winp and update Jenkins to it in next LTS?

          Ben Hines added a comment -

          We are also seeing this error pretty often.

          Ben Hines added a comment - We are also seeing this error pretty often.

          Oleg Nenashev added a comment -

          Additional fix has been integrated towards 2.34

          Oleg Nenashev added a comment - Additional fix has been integrated towards 2.34

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385
          Log:
          Update winp to 1.24. In particular, it addresses issues like JENKINS-20913(https://issues.jenkins-ci.org/browse/JENKINS-20913) (#2619)

              1. Changes to be picked
              1. 1.24

          Release date: Nov 2, 2016

              1. 1.23

          Release date: Fev 16, 2015

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385 Log: Update winp to 1.24. In particular, it addresses issues like JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) (#2619) Changes to be picked 1.24 Release date: Nov 2, 2016 Issue #22 ( https://github.com/kohsuke/winp/issues/22 ) - Winp sometimes kills random processes when using killRecursive. ( PR #23 ( https://github.com/kohsuke/winp/pull/23 )) [WINP-10] ( https://java.net/jira/browse/WINP-10 ) - Fix for `getCmdLineAndEnvVars()` which fails on x64 versions of Windows. ( PR #20 ( https://github.com/kohsuke/winp/pull/20 )) Issue #24 ( https://github.com/kohsuke/winp/issues/24 ) - Wrong folder when using the `winp.folder.preferred` system property (parent instead of the actual folder). ( PR #25 ( https://github.com/kohsuke/winp/pull/25 )) Issue #26 ( https://github.com/kohsuke/winp/issues/26 ), JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) - Native class now tries loading DLLs via the temp location. ( PR #27 ( https://github.com/kohsuke/winp/pull/27 )) 1.23 Release date: Fev 16, 2015 Migrate native components to Visual Studio Community 2013. ( PR #14 ( https://github.com/kohsuke/winp/pull/14 )) Provide a `winp.unpack.dll.to.parent.dir` property, which disables DLL unpacking to the parent dir. ( PR #14 ( https://github.com/kohsuke/winp/pull/12 ))

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/e66bfbafa99fc092c6f9fe51f8bf5be267340557
          Log:
          Update winp to 1.24. In particular, it addresses issues like JENKINS-20913(https://issues.jenkins-ci.org/browse/JENKINS-20913) (#2619)

              1. Changes to be picked
              1. 1.24

          Release date: Nov 2, 2016

              1. 1.23

          Release date: Fev 16, 2015

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/e66bfbafa99fc092c6f9fe51f8bf5be267340557 Log: Update winp to 1.24. In particular, it addresses issues like JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) (#2619) Changes to be picked 1.24 Release date: Nov 2, 2016 Issue #22 ( https://github.com/kohsuke/winp/issues/22 ) - Winp sometimes kills random processes when using killRecursive. ( PR #23 ( https://github.com/kohsuke/winp/pull/23 )) [WINP-10] ( https://java.net/jira/browse/WINP-10 ) - Fix for `getCmdLineAndEnvVars()` which fails on x64 versions of Windows. ( PR #20 ( https://github.com/kohsuke/winp/pull/20 )) Issue #24 ( https://github.com/kohsuke/winp/issues/24 ) - Wrong folder when using the `winp.folder.preferred` system property (parent instead of the actual folder). ( PR #25 ( https://github.com/kohsuke/winp/pull/25 )) Issue #26 ( https://github.com/kohsuke/winp/issues/26 ), JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) - Native class now tries loading DLLs via the temp location. ( PR #27 ( https://github.com/kohsuke/winp/pull/27 )) 1.23 Release date: Fev 16, 2015 Migrate native components to Visual Studio Community 2013. ( PR #14 ( https://github.com/kohsuke/winp/pull/14 )) Provide a `winp.unpack.dll.to.parent.dir` property, which disables DLL unpacking to the parent dir. ( PR #14 ( https://github.com/kohsuke/winp/pull/12 )) (cherry picked from commit 63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385)

            oleg_nenashev Oleg Nenashev
            oleg_nenashev Oleg Nenashev
            Votes:
            14 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: