Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20759

On windows slaves, occasional crash after successful build: Failed to write winp.x64.dll

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • core
    • None
    • Windows Server 2012 VM (amd64)
      Build via Maven 3.0.4 / 3.1.0
      JRE 7.0_25-b17

      Occasionally, remote windows slaves die with the following trace spit to the screen on the remote slave (I am running via command line, not as a service):

       
      Nov 20, 2013 4:16:10 PM org.jvnet.winp.Native load
      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)
              at java.io.FileOutputStream.open(Native Method)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
              at org.jvnet.winp.Native.load(Native.java:81)
              at org.jvnet.winp.Native.<clinit>(Native.java:52)
              at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
              at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:477)
              at hudson.util.ProcessTree.get(ProcessTree.java:336)
              at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
              at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
              at hudson.remoting.UserRequest.perform(UserRequest.java:118)
              at hudson.remoting.UserRequest.perform(UserRequest.java:48)
              at hudson.remoting.Request$2.run(Request.java:326)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
              at java.util.concurrent.FutureTask.run(FutureTask.java:166)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at hudson.remoting.Engine$1$1.run(Engine.java:63)
              at java.lang.Thread.run(Thread.java:724)
      
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000c8b3a90, pid=7136, tid=2024
      #
      # JRE version: 7.0_25-b17
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode windows-amd64 compressed oops)
      # Problematic frame:
      # C  0x000000000c8b3a90
      #
      # Core dump written. Default location: C:\\hs_err_pid7136.mdmp
      #
      # An error report file with more information is saved as:
      # C:\\hs_err_pid7136.log
      

      When this occurs, here's the trailing end of the build console for this build:

       
      ...
      ...
      ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 1:33.420s
      [INFO] Finished at: Wed Nov 20 16:16:09 EST 2013
      [INFO] Final Memory: 42M/272M
      [INFO] ------------------------------------------------------------------------
      [JENKINS] Archiving C:\jenkins\workspace\<redacted>\pom.xml to <redacted>SNAPSHOT.pom
      [JENKINS] Archiving C:\jenkins\workspace\<redacted>-SNAPSHOT.jar to <redacted>jar
      channel stopped
      FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      	at hudson.remoting.Request.call(Request.java:174)
      	at hudson.remoting.Channel.call(Channel.java:714)
      	at hudson.Launcher$RemoteLauncher.kill(Launcher.java:887)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:585)
      	at hudson.model.Run.execute(Run.java:1679)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:509)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:230)
      Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.Request.abort(Request.java:299)
      	at hudson.remoting.Channel.terminate(Channel.java:774)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
      Caused by: java.net.SocketException: Connection reset
      	at java.net.SocketInputStream.read(Unknown Source)
      	at java.net.SocketInputStream.read(Unknown Source)
      	at java.io.BufferedInputStream.fill(Unknown Source)
      	at java.io.BufferedInputStream.read(Unknown Source)
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
      	at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
      	at java.io.ObjectInputStream.readObject0(Unknown Source)
      	at java.io.ObjectInputStream.readObject(Unknown Source)
      	at hudson.remoting.Command.readFrom(Command.java:92)
      	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      

      The post build action that should have occurred at this point was the "Editable Email Notification" plugin. Not sure if it got that far or if the crash is immediately prior to that.

      This happens about 4-5 times out of every 100 builds. All of my build configs (for different projects, all maven builds, all executing the email-ext plugin afterwards...) do this.

      Occasionally, it will show the same stacktrace on the remote slave..

      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)

      .. but not crash.

      According to my google searches, a couple of people have been hitting this, though I couldn't find a JIRA issue logged against it. Not sure if anyone has determined root cause yet. I'll attach what information I can, though my company will frown upon me attaching the core dump, so I might not be able to do that.

          [JENKINS-20759] On windows slaves, occasional crash after successful build: Failed to write winp.x64.dll

          boris ivan added a comment -

          crashdump log file

          boris ivan added a comment - crashdump log file

          Oleg Nenashev added a comment -

          The issue seems to be very similar to JENKINS-20913.
          BTW, in my case JVM does not crash (1.7.0_u40)

          Oleg Nenashev added a comment - The issue seems to be very similar to JENKINS-20913 . BTW, in my case JVM does not crash (1.7.0_u40)

          Daniel Beck added a comment -

          Can this still be reproduced in recent versions of Jenkins? JENKINS-20913 indicates the issue this crash seems to result from has been resolved a few months ago.

          Daniel Beck added a comment - Can this still be reproduced in recent versions of Jenkins? JENKINS-20913 indicates the issue this crash seems to result from has been resolved a few months ago.

          boris ivan added a comment -

          I have not seen this problem for a several weeks now, though admittedly I can't pinpoint which release stopped it. (I tend to upgrade within the week of a release coming out, though I never know when I need to upgrade slave.jar... I upgrade that much less frequently). But the problem happened so frequently before, that it certainly seems to have been addressed.

          boris ivan added a comment - I have not seen this problem for a several weeks now, though admittedly I can't pinpoint which release stopped it. (I tend to upgrade within the week of a release coming out, though I never know when I need to upgrade slave.jar... I upgrade that much less frequently). But the problem happened so frequently before, that it certainly seems to have been addressed.

          Daniel Beck added a comment -

          Assuming resolved by the linked issue.

          If this occurs again in newer versions of Jenkins (no older than eight weeks), please file a new issue with then current information (stack trace, logs, etc.).

          Daniel Beck added a comment - Assuming resolved by the linked issue. If this occurs again in newer versions of Jenkins (no older than eight weeks), please file a new issue with then current information (stack trace, logs, etc.).

            Unassigned Unassigned
            borisivan boris ivan
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: