Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64082

Jenkins JNLP agent crashes almost every hour due to JVM crash

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: core, remoting
    • Environment:
      Master: Windows Server 2012 R2 Standard, Build 9600, Jenkins 2.249.2, AdoptOpenJDK\jre-8.0.265.01-hotspot
      Agent: Windows Server 2019 Standard, Build 17763, AdoptOpenJDK\jre-8.0.265.01-hotspot
    • Similar Issues:

      Description

      Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer. 

      When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason. 

      #
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
      #
      # JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
      # Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
      # Problematic frame:
      # V  [jvm.dll+0x1ed41b]
      #
      # Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.)
      #
      # If you would like to submit a bug report, please visit:
      #   http://bugreport.java.com/bugreport/crash.jsp
      #
      
      ---------------  T H R E A D  ---------------
      Current thread (0x0000012702413800):  WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
      siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010 
      
      Registers:
      RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
      RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
      R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
      R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
      RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202
      
      Top of Stack: (sp=0x0000001b501ffb70)
      0x0000001b501ffb70:   0000012702413800 0000000064c02cd1
      0x0000001b501ffb80:   0000000000000001 0000000064c021a5
      0x0000001b501ffb90:   0000000000000000 0000000064c260bf
      0x0000001b501ffba0:   00005593d9fd7114 0000000000000002
      0x0000001b501ffbb0:   0000001b501ffe10 0000000064c32926
      0x0000001b501ffbc0:   00000000651fc948 0000000064c2ac9d
      0x0000001b501ffbd0:   0000000000000001 000001274a4c58d0
      0x0000001b501ffbe0:   0000000000000000 0000000000000000
      0x0000001b501ffbf0:   0000000000000000 0000000000000000
      0x0000001b501ffc00:   0000000000000002 0000000064c32170
      0x0000001b501ffc10:   0000001b501ffe10 000001277fb2ab30
      0x0000001b501ffc20:   0000012702413800 0000000000000000
      0x0000001b501ffc30:   0000000000000000 0000000000000000
      0x0000001b501ffc40:   0000000000000000 0000000000000000
      0x0000001b501ffc50:   0000000000000000 0000000000000000
      0x0000001b501ffc60:   000001277fb2ab30 0000000064c83c6a 
      
      Instructions: (pc=0x0000000064c0d41b)
      0x0000000064c0d3fb:   5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
      0x0000000064c0d40b:   49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
      0x0000000064c0d41b:   48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
      0x0000000064c0d42b:   cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48 
      
      Register to memory mapping:
      RAX=0x0000000009976d58 is an unknown value
      RBX=0x000001274a4d3170 is an unknown value
      RCX=0x000001274acb1420 is an unknown value
      RDX=0x0000000000000033 is an unknown value
      RSP=0x0000001b501ffb70 is an unknown value
      RBP=0x0000000000000002 is an unknown value
      RSI=0x000001274a4b5690 is an unknown value
      RDI=0x0000000000000000 is an unknown value
      R8 =0x0000000000000031 is an unknown value
      R9 =0x00000127023e4983 is an unknown value
      R10=0x00000000546c6148 is an unknown value
      R11=0xffffffffb9e4ccfa is an unknown value
      R12=0x0000000000000000 is an unknown value
      R13=0x0000000000000000 is an unknown value
      R14=0x0000000000000002 is an unknown value
      R15=0x0000000000000000 is an unknown value
      
      Stack: [0x0000001b50100000,0x0000001b50200000],  sp=0x0000001b501ffb70,  free space=1022k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V  [jvm.dll+0x1ed41b]
      V  [jvm.dll+0x2060bf]
      V  [jvm.dll+0x20ac9d]
      V  [jvm.dll+0x212170]
      V  [jvm.dll+0x263c6a]
      C  [msvcr120.dll+0x24f7f]
      C  [msvcr120.dll+0x25126]
      C  [KERNEL32.DLL+0x17974]
      C  [ntdll.dll+0x6a271]
      
      ---------------  P R O C E S S  ---------------
      Java Threads: ( => current thread )
        0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
        0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
        0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
        0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
        0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
        0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
        0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
        0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
        0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
        0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)]  0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
        0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
        0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
        0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
        0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
        0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
        0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
        0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
        0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
        0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
        0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
        0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
        0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
        0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
        0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
        0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
        0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
        0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
        0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
        0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
        0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]
      
      Other Threads:
        0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
      =>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
      
      VM state:not at safepoint (normal execution)
      
      VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
      [0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
      [0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800
      
      heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
      Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
      Compressed class space size: 1073741824 Address: 0x00000007c0000000
      
      Heap:
       PSYoungGen      total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
        eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
        from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
        to   space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
       ParOldGen       total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
        object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
       Metaspace       used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
        class space    used 2025K, capacity 2173K, committed 2176K, reserved 1048576K
      
      Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000
      
      Marking Bits: (ParMarkBitMap*) 0x0000000065207790
       Begin Bits: [0x000001275d570000, 0x000001276d360000)
       End Bits:   [0x000001276d360000, 0x000001277d150000)
      
      Polling page: 0x000001274ac90000
      
      CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
       bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
       total_blobs=2958 nmethods=2514 adapters=354
       compilation: enabled
      
      Compilation events (10 events):
      Event: 3581.232 Thread 0x0000012702356800 2988   !   3       hudson.remoting.Channel$1::handle (233 bytes)
      Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
      Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
      Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
      Event: 3582.161 Thread 0x000001274a58d000 2989       4       java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
      Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
      Event: 3602.216 Thread 0x0000012702355000 2990   !   4       java.io.ObjectStreamClass::getReflector (336 bytes)
      Event: 3602.237 Thread 0x000001274a58a000 2991       4       java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
      Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]
      
      GC Heap History (2 events):
      Event: 242.224 GC heap before
      {Heap before GC invocations=1 (full 0):
       PSYoungGen      total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
        eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
        from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
        to   space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
       ParOldGen       total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)  object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
       Metaspace       used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
        class space    used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
      Event: 242.238 GC heap after
      Heap after GC invocations=1 (full 0):
       PSYoungGen      total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)  eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
        from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
        to   space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
       ParOldGen       total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)  object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
       Metaspace       used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
        class space    used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
      }
      
      Deoptimization events (10 events):
      Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
      Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
      Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
      Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
      Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
      Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
      Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
      Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
      Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
      Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35
      
      Classes redefined (0 events):
      No events
      
      Internal exceptions (10 events):
      Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
      Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
      
      Events (10 events):
      Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
      Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
      Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
      Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
      Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
      Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
      Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710
      
      Dynamic libraries:
      0x00007ff7ebb00000 - 0x00007ff7ebb3a000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
      0x00007ff96bad0000 - 0x00007ff96bcbd000  C:\Windows\SYSTEM32\ntdll.dll
      0x00007ff96acc0000 - 0x00007ff96ad73000  C:\Windows\System32\KERNEL32.DLL
      0x00007ff9686d0000 - 0x00007ff968965000  C:\Windows\System32\KERNELBASE.dll
      0x00007ff96b9f0000 - 0x00007ff96ba93000  C:\Windows\System32\ADVAPI32.dll
      0x00007ff9694a0000 - 0x00007ff96953e000  C:\Windows\System32\msvcrt.dll
      0x00007ff96b0b0000 - 0x00007ff96b14e000  C:\Windows\System32\sechost.dll
      0x00007ff96ab90000 - 0x00007ff96acb2000  C:\Windows\System32\RPCRT4.dll
      0x00007ff968f50000 - 0x00007ff9690e7000  C:\Windows\System32\USER32.dll
      0x00007ff967d70000 - 0x00007ff967d90000  C:\Windows\System32\win32u.dll
      0x00007ff969540000 - 0x00007ff969569000  C:\Windows\System32\GDI32.dll
      0x00007ff967d90000 - 0x00007ff967f2a000  C:\Windows\System32\gdi32full.dll
      0x00007ff968c00000 - 0x00007ff968ca0000  C:\Windows\System32\msvcp_win.dll
      0x00007ff968970000 - 0x00007ff968a6a000  C:\Windows\System32\ucrtbase.dll
      0x00007ff957030000 - 0x00007ff9572a9000  C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
      0x00007ff96ad80000 - 0x00007ff96b0ac000  C:\Windows\System32\combase.dll
      0x00007ff968a70000 - 0x00007ff968aee000  C:\Windows\System32\bcryptPrimitives.dll
      0x00007ff93d280000 - 0x00007ff93d36f000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
      0x00007ff944dc0000 - 0x00007ff944e66000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
      0x0000000064a20000 - 0x0000000065286000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
      0x00007ff96b4a0000 - 0x00007ff96b4a8000  C:\Windows\System32\PSAPI.DLL
      0x00007ff965d70000 - 0x00007ff965d94000  C:\Windows\SYSTEM32\WINMM.dll
      0x00007ff95e840000 - 0x00007ff95e84a000  C:\Windows\SYSTEM32\VERSION.dll
      0x00007ff965db0000 - 0x00007ff965db9000  C:\Windows\SYSTEM32\WSOCK32.dll
      0x00007ff9695f0000 - 0x00007ff96965d000  C:\Windows\System32\WS2_32.dll
      0x00007ff965d40000 - 0x00007ff965d6d000  C:\Windows\SYSTEM32\winmmbase.dll
      0x00007ff968680000 - 0x00007ff9686ca000  C:\Windows\System32\cfgmgr32.dll
      0x00007ff965af0000 - 0x00007ff965aff000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
      0x00007ff965a50000 - 0x00007ff965a78000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
      0x00007ff954270000 - 0x00007ff954286000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
      0x00007ff969690000 - 0x00007ff96ab88000  C:\Windows\System32\SHELL32.dll
      0x00007ff968cf0000 - 0x00007ff968d98000  C:\Windows\System32\shcore.dll
      0x00007ff967f30000 - 0x00007ff96867f000  C:\Windows\System32\windows.storage.dll
      0x00007ff967ac0000 - 0x00007ff967ae4000  C:\Windows\System32\profapi.dll
      0x00007ff967af0000 - 0x00007ff967b4d000  C:\Windows\System32\powrprof.dll
      0x00007ff96b510000 - 0x00007ff96b562000  C:\Windows\System32\shlwapi.dll
      0x00007ff967b50000 - 0x00007ff967b61000  C:\Windows\System32\kernel.appcore.dll
      0x00007ff968ca0000 - 0x00007ff968cb7000  C:\Windows\System32\cryptsp.dll
      0x00007ff953ff0000 - 0x00007ff954013000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
      0x00007ff966e40000 - 0x00007ff966e73000  C:\Windows\system32\rsaenh.dll
      0x00007ff968cc0000 - 0x00007ff968ce6000  C:\Windows\System32\bcrypt.dll
      0x00007ff9679f0000 - 0x00007ff967a18000  C:\Windows\SYSTEM32\USERENV.dll
      0x00007ff967490000 - 0x00007ff96749c000  C:\Windows\SYSTEM32\CRYPTBASE.dll
      0x00007ff953f10000 - 0x00007ff953f2a000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
      0x00007ff9672c0000 - 0x00007ff967327000  C:\Windows\system32\mswsock.dll
      0x00007ff966fe0000 - 0x00007ff96701d000  C:\Windows\SYSTEM32\IPHLPAPI.DLL
      0x00007ff969660000 - 0x00007ff969668000  C:\Windows\System32\NSI.dll
      0x00007ff9634d0000 - 0x00007ff9634e6000  C:\Windows\SYSTEM32\dhcpcsvc6.DLL
      0x00007ff9634b0000 - 0x00007ff9634cc000  C:\Windows\SYSTEM32\dhcpcsvc.DLL
      0x00007ff967020000 - 0x00007ff9670e7000  C:\Windows\SYSTEM32\DNSAPI.dll
      0x00007ff953e80000 - 0x00007ff953e92000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
      0x00007ff965ae0000 - 0x00007ff965aec000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
      0x00007ff967b90000 - 0x00007ff967d6c000  C:\Windows\System32\CRYPT32.dll
      0x00007ff967b70000 - 0x00007ff967b82000  C:\Windows\System32\MSASN1.dll
      0x00007ff967590000 - 0x00007ff9675b6000  C:\Windows\SYSTEM32\ncrypt.dll
      0x00007ff967550000 - 0x00007ff96758c000  C:\Windows\SYSTEM32\NTASN1.dll
      0x00007ff960140000 - 0x00007ff96014a000  C:\Windows\System32\rasadhlp.dll
      0x00007ff963600000 - 0x00007ff963679000  C:\Windows\System32\fwpuclnt.dll
      0x00007ff939d30000 - 0x00007ff939ec5000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
      0x00007ff968da0000 - 0x00007ff968e64000  C:\Windows\System32\OLEAUT32.dll
      0x00007ff965c20000 - 0x00007ff965cac000  C:\Windows\SYSTEM32\apphelp.dll
      0x00007ff94c620000 - 0x00007ff94c662000  D:\Temp\jna--1366022967\jna4539908599793538979.dll
      0x00007ff960b50000 - 0x00007ff960d3d000  C:\Windows\system32\DBGHELP.DLL
      0x00007ff9606b0000 - 0x00007ff9606da000  C:\Windows\SYSTEM32\dbgcore.DLL
      
      VM Arguments:
      jvm_args: -Xrs -DSoftKillWaitSeconds=0 
      java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
      Launcher Type: SUN_STANDARD
      
      Environment Variables:
      JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
      PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
      USERNAME=BuildSrv
      OS=Windows_NT
      PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD
      
      ---------------  S Y S T E M  ---------------
      
      OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)
      
      CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1
      
      Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)
      
      vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)
      
      time: Thu Oct 29 17:55:26 2020
      timezone: Pacific Daylight Time
      elapsed time: 3646 seconds (0d 1h 0m 46s)

       

      On the agent:

      Inbound agent connected from internal_fdqn/internal_ip:64539
      Inbound agent connected from internal_fdqn/internal_ip:64539
      Remoting version: 4.5
      This is a Windows agentAgent successfully connected and online
      
      ERROR: Connection terminatedjava.nio.channels.ClosedChannelException 
          at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) 
          at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179) 
          at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) 
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 
          at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) 
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
          at java.lang.Thread.run(Thread.java:748)

      On the master:

      2020-10-30 02:48:54.152+0000 [id=2405]	INFO	j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
      2020-10-30 02:48:55.895+0000 [id=2414]	INFO	h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
      2020-10-30 02:48:55.931+0000 [id=2415]	INFO	h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261

      It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

       

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          Hint on stackoverflow is a possible corruption of the Java installation on Windows.

          https://stackoverflow.com/questions/44327267/java-exception-in-page-error

          https://stackoverflow.com/questions/44327267/java-exception-in-page-error suggests it could be a hardware fault.

          Show
          markewaite Mark Waite added a comment - Hint on stackoverflow is a possible corruption of the Java installation on Windows. https://stackoverflow.com/questions/44327267/java-exception-in-page-error https://stackoverflow.com/questions/44327267/java-exception-in-page-error suggests it could be a hardware fault.
          Hide
          blee4 Bryant Lee added a comment -

          Mark Waite

          I've tried with multiple different Java and got the same results. I've tried with AdoptOpenJDK 8 and 11. I've tried with older Oracle JRE 8. OpenJDK 8. All with the same result. We are experiencing this same exact issue across 4 of our new server machines. So I doubt it's a disk issue. The only major difference between the old and new machines is the processor. Old machines have various types of Intels, whereas the new machines have AMD EPYC 7402P in them. 

          Show
          blee4 Bryant Lee added a comment - Mark Waite I've tried with multiple different Java and got the same results. I've tried with AdoptOpenJDK 8 and 11. I've tried with older Oracle JRE 8. OpenJDK 8. All with the same result. We are experiencing this same exact issue across 4 of our new server machines. So I doubt it's a disk issue. The only major difference between the old and new machines is the processor. Old machines have various types of Intels, whereas the new machines have AMD EPYC 7402P in them. 

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            blee4 Bryant Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: