[JENKINS-64082] Jenkins JNLP agent crashes almost every hour due to JVM crash

Type: Bug
Resolution: Cannot Reproduce
Priority: Blocker
Component/s: core, remoting
Labels:
- jenkins
- slave
- windows
Environment:
Master: Windows Server 2012 R2 Standard, Build 9600, Jenkins 2.249.2, AdoptOpenJDK\jre-8.0.265.01-hotspot
Agent: Windows Server 2019 Standard, Build 17763, AdoptOpenJDK\jre-8.0.265.01-hotspot

Similar Issues:
Powered by SuggestiMate

Show

Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.

#
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------
Current thread (0x0000012702413800):  WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010 

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70:   0000012702413800 0000000064c02cd1
0x0000001b501ffb80:   0000000000000001 0000000064c021a5
0x0000001b501ffb90:   0000000000000000 0000000064c260bf
0x0000001b501ffba0:   00005593d9fd7114 0000000000000002
0x0000001b501ffbb0:   0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0:   00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0:   0000000000000001 000001274a4c58d0
0x0000001b501ffbe0:   0000000000000000 0000000000000000
0x0000001b501ffbf0:   0000000000000000 0000000000000000
0x0000001b501ffc00:   0000000000000002 0000000064c32170
0x0000001b501ffc10:   0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20:   0000012702413800 0000000000000000
0x0000001b501ffc30:   0000000000000000 0000000000000000
0x0000001b501ffc40:   0000000000000000 0000000000000000
0x0000001b501ffc50:   0000000000000000 0000000000000000
0x0000001b501ffc60:   000001277fb2ab30 0000000064c83c6a 

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb:   5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b:   49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b:   48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b:   cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48 

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000],  sp=0x0000001b501ffb70,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x1ed41b]
V  [jvm.dll+0x2060bf]
V  [jvm.dll+0x20ac9d]
V  [jvm.dll+0x212170]
V  [jvm.dll+0x263c6a]
C  [msvcr120.dll+0x24f7f]
C  [msvcr120.dll+0x25126]
C  [KERNEL32.DLL+0x17974]
C  [ntdll.dll+0x6a271]

---------------  P R O C E S S  ---------------
Java Threads: ( => current thread )
  0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
  0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
  0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
  0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
  0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
  0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
  0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
  0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
  0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
  0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)]  0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
  0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
  0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
  0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
  0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
  0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
  0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
  0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
  0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
  0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
  0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
  0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
  0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
  0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
  0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
  0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
  0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
  0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
  0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
  0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
  0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
  0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
 PSYoungGen      total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
  eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
  from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
  to   space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
 ParOldGen       total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
  object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
 Metaspace       used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
  class space    used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
 Begin Bits: [0x000001275d570000, 0x000001276d360000)
 End Bits:   [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
 bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
 total_blobs=2958 nmethods=2514 adapters=354
 compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988   !   3       hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989       4       java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990   !   4       java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991       4       java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
 PSYoungGen      total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
  eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
  from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
  to   space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
 ParOldGen       total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)  object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
 Metaspace       used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
  class space    used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
 PSYoungGen      total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)  eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
  from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
  to   space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
 ParOldGen       total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)  object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
 Metaspace       used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
  class space    used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000  C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000  C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000  C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000  C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000  C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000  C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000  C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000  C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000  C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000  C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000  C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000  C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000  C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000  C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000  C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000  C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000  C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000  C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000  C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000  C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000  C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000  C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000  C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000  C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000  C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000  C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000  C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000  C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000  C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000  C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000  C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000  C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000  C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000  C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000  C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000  C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000  C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000  C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000  C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000  C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000  C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000  C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000  C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000  C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000  C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000  C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000  C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000  C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000  C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000  C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000  D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000  C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000  C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0 
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

---------------  S Y S T E M  ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s)

On the agent:

Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException 
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) 
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179) 
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) 
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
    at java.lang.Thread.run(Thread.java:748)

On the master:

2020-10-30 02:48:54.152+0000 [id=2405]	INFO	j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414]	INFO	h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415]	INFO	h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261

It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

Bryant Lee created issue - 2020-10-30 08:22

Bryant Lee made changes - 2020-10-30 08:24

Description

Original: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.

{noformat}
### A fatal error has been detected by the Java Runtime Environment:## EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c## JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)# Problematic frame:# V [jvm.dll+0x1ed41b]## Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)## If you would like to submit a bug report, please visit:# http://bugreport.java.com/bugreport/crash.jsp#
--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010
Registers:RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000R8 =0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfaR12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202
Top of Stack: (sp=0x0000001b501ffb70)0x0000001b501ffb70: 0000012702413800 0000000064c02cd10x0000001b501ffb80: 0000000000000001 0000000064c021a50x0000001b501ffb90: 0000000000000000 0000000064c260bf0x0000001b501ffba0: 00005593d9fd7114 00000000000000020x0000001b501ffbb0: 0000001b501ffe10 0000000064c329260x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d0x0000001b501ffbd0: 0000000000000001 000001274a4c58d00x0000001b501ffbe0: 0000000000000000 00000000000000000x0000001b501ffbf0: 0000000000000000 00000000000000000x0000001b501ffc00: 0000000000000002 0000000064c321700x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab300x0000001b501ffc20: 0000012702413800 00000000000000000x0000001b501ffc30: 0000000000000000 00000000000000000x0000001b501ffc40: 0000000000000000 00000000000000000x0000001b501ffc50: 0000000000000000 00000000000000000x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a
Instructions: (pc=0x0000000064c0d41b)0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 280x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48 Register to memory mapping:
RAX=0x0000000009976d58 is an unknown valueRBX=0x000001274a4d3170 is an unknown valueRCX=0x000001274acb1420 is an unknown valueRDX=0x0000000000000033 is an unknown valueRSP=0x0000001b501ffb70 is an unknown valueRBP=0x0000000000000002 is an unknown valueRSI=0x000001274a4b5690 is an unknown valueRDI=0x0000000000000000 is an unknown valueR8 =0x0000000000000031 is an unknown valueR9 =0x00000127023e4983 is an unknown valueR10=0x00000000546c6148 is an unknown valueR11=0xffffffffb9e4ccfa is an unknown valueR12=0x0000000000000000 is an unknown valueR13=0x0000000000000000 is an unknown valueR14=0x0000000000000002 is an unknown valueR15=0x0000000000000000 is an unknown valueStack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)V [jvm.dll+0x1ed41b]V [jvm.dll+0x2060bf]V [jvm.dll+0x20ac9d]V [jvm.dll+0x212170]V [jvm.dll+0x263c6a]C [msvcr120.dll+0x24f7f]C [msvcr120.dll+0x25126]C [KERNEL32.DLL+0x17974]C [ntdll.dll+0x6a271]--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)] 0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)] 0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)] 0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)] 0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)] 0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)] 0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)] 0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)] 0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)] 0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)] 0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)] 0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)] 0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)] 0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)] 0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)] 0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)] 0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)] 0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)] 0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)] 0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)] 0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)] 0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)] 0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)] 0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)] 0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)] 0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)] 0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)] 0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)] 0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)] 0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]
Other Threads: 0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800
heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3Narrow klass base: 0x0000000000000000, Narrow klass shift: 3Compressed class space size: 1073741824 Address: 0x00000007c0000000
Heap: PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K
Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000
Marking Bits: (ParMarkBitMap*) 0x0000000065207790 Begin Bits: [0x000001275d570000, 0x000001276d360000) End Bits: [0x000001276d360000, 0x000001277d150000)
Polling page: 0x000001274ac90000
CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000] total_blobs=2958 nmethods=2514 adapters=354 compilation: enabled
Compilation events (10 events):Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]
GC Heap History (2 events):Event: 242.224 GC heap before{Heap before GC invocations=1 (full 0): PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000) from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000) ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576KEvent: 242.238 GC heap afterHeap after GC invocations=1 (full 0): PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K}
Deoptimization events (10 events):Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransfeEvent: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35
Classes redefined (0 events):No events
Internal exceptions (10 events):Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Events (10 events):Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias doneEvent: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710Dynamic libraries:0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL
VM Arguments:jvm_args: -Xrs -DSoftKillWaitSeconds=0 java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jarLauncher Type: SUN_STANDARD
Environment Variables:JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsAppsUSERNAME=BuildSrvOS=Windows_NTPROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD
--------------- S Y S T E M ---------------
OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)
CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1
Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)
vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)
time: Thu Oct 29 17:55:26 2020timezone: Pacific Daylight Timeelapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:

{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539Inbound agent connected from internal_fdqn/internal_ip:64539Remoting version: 4.5This is a Windows agentAgent successfully connected and online
ERROR: Connection terminatedjava.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

New: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
### A fatal error has been detected by the Java Runtime Environment:## EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c## JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)# Problematic frame:# V [jvm.dll+0x1ed41b]## Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)## If you would like to submit a bug report, please visit:# http://bugreport.java.com/bugreport/crash.jsp#
--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010
Registers:RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000R8 =0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfaR12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202
Top of Stack: (sp=0x0000001b501ffb70)0x0000001b501ffb70: 0000012702413800 0000000064c02cd10x0000001b501ffb80: 0000000000000001 0000000064c021a50x0000001b501ffb90: 0000000000000000 0000000064c260bf0x0000001b501ffba0: 00005593d9fd7114 00000000000000020x0000001b501ffbb0: 0000001b501ffe10 0000000064c329260x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d0x0000001b501ffbd0: 0000000000000001 000001274a4c58d00x0000001b501ffbe0: 0000000000000000 00000000000000000x0000001b501ffbf0: 0000000000000000 00000000000000000x0000001b501ffc00: 0000000000000002 0000000064c321700x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab300x0000001b501ffc20: 0000012702413800 00000000000000000x0000001b501ffc30: 0000000000000000 00000000000000000x0000001b501ffc40: 0000000000000000 00000000000000000x0000001b501ffc50: 0000000000000000 00000000000000000x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a
Instructions: (pc=0x0000000064c0d41b)0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 280x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48 Register to memory mapping:
RAX=0x0000000009976d58 is an unknown valueRBX=0x000001274a4d3170 is an unknown valueRCX=0x000001274acb1420 is an unknown valueRDX=0x0000000000000033 is an unknown valueRSP=0x0000001b501ffb70 is an unknown valueRBP=0x0000000000000002 is an unknown valueRSI=0x000001274a4b5690 is an unknown valueRDI=0x0000000000000000 is an unknown valueR8 =0x0000000000000031 is an unknown valueR9 =0x00000127023e4983 is an unknown valueR10=0x00000000546c6148 is an unknown valueR11=0xffffffffb9e4ccfa is an unknown valueR12=0x0000000000000000 is an unknown valueR13=0x0000000000000000 is an unknown valueR14=0x0000000000000002 is an unknown valueR15=0x0000000000000000 is an unknown valueStack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)V [jvm.dll+0x1ed41b]V [jvm.dll+0x2060bf]V [jvm.dll+0x20ac9d]V [jvm.dll+0x212170]V [jvm.dll+0x263c6a]C [msvcr120.dll+0x24f7f]C [msvcr120.dll+0x25126]C [KERNEL32.DLL+0x17974]C [ntdll.dll+0x6a271]--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)] 0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)] 0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)] 0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)] 0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)] 0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)] 0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)] 0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)] 0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)] 0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)] 0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)] 0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)] 0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)] 0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)] 0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)] 0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)] 0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)] 0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)] 0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)] 0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)] 0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)] 0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)] 0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)] 0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)] 0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)] 0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)] 0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)] 0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)] 0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)] 0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]
Other Threads: 0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800
heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3Narrow klass base: 0x0000000000000000, Narrow klass shift: 3Compressed class space size: 1073741824 Address: 0x00000007c0000000
Heap: PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K
Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000
Marking Bits: (ParMarkBitMap*) 0x0000000065207790 Begin Bits: [0x000001275d570000, 0x000001276d360000) End Bits: [0x000001276d360000, 0x000001277d150000)
Polling page: 0x000001274ac90000
CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000] total_blobs=2958 nmethods=2514 adapters=354 compilation: enabled
Compilation events (10 events):Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]
GC Heap History (2 events):Event: 242.224 GC heap before{Heap before GC invocations=1 (full 0): PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000) from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000) ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576KEvent: 242.238 GC heap afterHeap after GC invocations=1 (full 0): PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K}
Deoptimization events (10 events):Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransfeEvent: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35
Classes redefined (0 events):No events
Internal exceptions (10 events):Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Events (10 events):Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias doneEvent: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710Dynamic libraries:0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL
VM Arguments:jvm_args: -Xrs -DSoftKillWaitSeconds=0 java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jarLauncher Type: SUN_STANDARD
Environment Variables:JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsAppsUSERNAME=BuildSrvOS=Windows_NTPROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD
--------------- S Y S T E M ---------------
OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)
CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1
Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)
vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)
time: Thu Oct 29 17:55:26 2020timezone: Pacific Daylight Timeelapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539Inbound agent connected from internal_fdqn/internal_ip:64539Remoting version: 4.5This is a Windows agentAgent successfully connected and online
ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

Bryant Lee made changes - 2020-10-30 08:49

Description

Original: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
### A fatal error has been detected by the Java Runtime Environment:## EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c## JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)# Problematic frame:# V [jvm.dll+0x1ed41b]## Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)## If you would like to submit a bug report, please visit:# http://bugreport.java.com/bugreport/crash.jsp#
--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010
Registers:RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000R8 =0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfaR12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202
Top of Stack: (sp=0x0000001b501ffb70)0x0000001b501ffb70: 0000012702413800 0000000064c02cd10x0000001b501ffb80: 0000000000000001 0000000064c021a50x0000001b501ffb90: 0000000000000000 0000000064c260bf0x0000001b501ffba0: 00005593d9fd7114 00000000000000020x0000001b501ffbb0: 0000001b501ffe10 0000000064c329260x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d0x0000001b501ffbd0: 0000000000000001 000001274a4c58d00x0000001b501ffbe0: 0000000000000000 00000000000000000x0000001b501ffbf0: 0000000000000000 00000000000000000x0000001b501ffc00: 0000000000000002 0000000064c321700x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab300x0000001b501ffc20: 0000012702413800 00000000000000000x0000001b501ffc30: 0000000000000000 00000000000000000x0000001b501ffc40: 0000000000000000 00000000000000000x0000001b501ffc50: 0000000000000000 00000000000000000x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a
Instructions: (pc=0x0000000064c0d41b)0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 280x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48 Register to memory mapping:
RAX=0x0000000009976d58 is an unknown valueRBX=0x000001274a4d3170 is an unknown valueRCX=0x000001274acb1420 is an unknown valueRDX=0x0000000000000033 is an unknown valueRSP=0x0000001b501ffb70 is an unknown valueRBP=0x0000000000000002 is an unknown valueRSI=0x000001274a4b5690 is an unknown valueRDI=0x0000000000000000 is an unknown valueR8 =0x0000000000000031 is an unknown valueR9 =0x00000127023e4983 is an unknown valueR10=0x00000000546c6148 is an unknown valueR11=0xffffffffb9e4ccfa is an unknown valueR12=0x0000000000000000 is an unknown valueR13=0x0000000000000000 is an unknown valueR14=0x0000000000000002 is an unknown valueR15=0x0000000000000000 is an unknown valueStack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)V [jvm.dll+0x1ed41b]V [jvm.dll+0x2060bf]V [jvm.dll+0x20ac9d]V [jvm.dll+0x212170]V [jvm.dll+0x263c6a]C [msvcr120.dll+0x24f7f]C [msvcr120.dll+0x25126]C [KERNEL32.DLL+0x17974]C [ntdll.dll+0x6a271]--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)] 0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)] 0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)] 0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)] 0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)] 0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)] 0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)] 0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)] 0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)] 0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)] 0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)] 0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)] 0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)] 0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)] 0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)] 0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)] 0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)] 0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)] 0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)] 0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)] 0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)] 0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)] 0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)] 0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)] 0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)] 0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)] 0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)] 0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)] 0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)] 0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]
Other Threads: 0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800
heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3Narrow klass base: 0x0000000000000000, Narrow klass shift: 3Compressed class space size: 1073741824 Address: 0x00000007c0000000
Heap: PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K
Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000
Marking Bits: (ParMarkBitMap*) 0x0000000065207790 Begin Bits: [0x000001275d570000, 0x000001276d360000) End Bits: [0x000001276d360000, 0x000001277d150000)
Polling page: 0x000001274ac90000
CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000] total_blobs=2958 nmethods=2514 adapters=354 compilation: enabled
Compilation events (10 events):Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]
GC Heap History (2 events):Event: 242.224 GC heap before{Heap before GC invocations=1 (full 0): PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000) from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000) ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576KEvent: 242.238 GC heap afterHeap after GC invocations=1 (full 0): PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000) from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000) to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000) ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000) Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K}
Deoptimization events (10 events):Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransfeEvent: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35
Classes redefined (0 events):No events
Internal exceptions (10 events):Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Events (10 events):Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias doneEvent: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias doneEvent: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710Dynamic libraries:0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL
VM Arguments:jvm_args: -Xrs -DSoftKillWaitSeconds=0 java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jarLauncher Type: SUN_STANDARD
Environment Variables:JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsAppsUSERNAME=BuildSrvOS=Windows_NTPROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD
--------------- S Y S T E M ---------------
OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)
CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1
Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)
vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)
time: Thu Oct 29 17:55:26 2020timezone: Pacific Daylight Timeelapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539Inbound agent connected from internal_fdqn/internal_ip:64539Remoting version: 4.5This is a Windows agentAgent successfully connected and online
ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

New: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70: 0000012702413800 0000000064c02cd1
0x0000001b501ffb80: 0000000000000001 0000000064c021a5
0x0000001b501ffb90: 0000000000000000 0000000064c260bf
0x0000001b501ffba0: 00005593d9fd7114 0000000000000002
0x0000001b501ffbb0: 0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0: 0000000000000001 000001274a4c58d0
0x0000001b501ffbe0: 0000000000000000 0000000000000000
0x0000001b501ffbf0: 0000000000000000 0000000000000000
0x0000001b501ffc00: 0000000000000002 0000000064c32170
0x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20: 0000012702413800 0000000000000000
0x0000001b501ffc30: 0000000000000000 0000000000000000
0x0000001b501ffc40: 0000000000000000 0000000000000000
0x0000001b501ffc50: 0000000000000000 0000000000000000
0x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1ed41b]
V [jvm.dll+0x2060bf]
V [jvm.dll+0x20ac9d]
V [jvm.dll+0x212170]
V [jvm.dll+0x263c6a]
C [msvcr120.dll+0x24f7f]
C [msvcr120.dll+0x25126]
C [KERNEL32.DLL+0x17974]
C [ntdll.dll+0x6a271]

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
Begin Bits: [0x000001275d570000, 0x000001276d360000)
End Bits: [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
total_blobs=2958 nmethods=2514 adapters=354
compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

--------------- S Y S T E M ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

Bryant Lee made changes - 2020-10-30 08:51

Description

Original: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70: 0000012702413800 0000000064c02cd1
0x0000001b501ffb80: 0000000000000001 0000000064c021a5
0x0000001b501ffb90: 0000000000000000 0000000064c260bf
0x0000001b501ffba0: 00005593d9fd7114 0000000000000002
0x0000001b501ffbb0: 0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0: 0000000000000001 000001274a4c58d0
0x0000001b501ffbe0: 0000000000000000 0000000000000000
0x0000001b501ffbf0: 0000000000000000 0000000000000000
0x0000001b501ffc00: 0000000000000002 0000000064c32170
0x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20: 0000012702413800 0000000000000000
0x0000001b501ffc30: 0000000000000000 0000000000000000
0x0000001b501ffc40: 0000000000000000 0000000000000000
0x0000001b501ffc50: 0000000000000000 0000000000000000
0x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1ed41b]
V [jvm.dll+0x2060bf]
V [jvm.dll+0x20ac9d]
V [jvm.dll+0x212170]
V [jvm.dll+0x263c6a]
C [msvcr120.dll+0x24f7f]
C [msvcr120.dll+0x25126]
C [KERNEL32.DLL+0x17974]
C [ntdll.dll+0x6a271]

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
Begin Bits: [0x000001275d570000, 0x000001276d360000)
End Bits: [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
total_blobs=2958 nmethods=2514 adapters=354
compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

--------------- S Y S T E M ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

New: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70: 0000012702413800 0000000064c02cd1
0x0000001b501ffb80: 0000000000000001 0000000064c021a5
0x0000001b501ffb90: 0000000000000000 0000000064c260bf
0x0000001b501ffba0: 00005593d9fd7114 0000000000000002
0x0000001b501ffbb0: 0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0: 0000000000000001 000001274a4c58d0
0x0000001b501ffbe0: 0000000000000000 0000000000000000
0x0000001b501ffbf0: 0000000000000000 0000000000000000
0x0000001b501ffc00: 0000000000000002 0000000064c32170
0x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20: 0000012702413800 0000000000000000
0x0000001b501ffc30: 0000000000000000 0000000000000000
0x0000001b501ffc40: 0000000000000000 0000000000000000
0x0000001b501ffc50: 0000000000000000 0000000000000000
0x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1ed41b]
V [jvm.dll+0x2060bf]
V [jvm.dll+0x20ac9d]
V [jvm.dll+0x212170]
V [jvm.dll+0x263c6a]
C [msvcr120.dll+0x24f7f]
C [msvcr120.dll+0x25126]
C [KERNEL32.DLL+0x17974]
C [ntdll.dll+0x6a271]

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
Begin Bits: [0x000001275d570000, 0x000001276d360000)
End Bits: [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
total_blobs=2958 nmethods=2514 adapters=354
compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

--------------- S Y S T E M ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

Bryant Lee made changes - 2020-10-30 08:53

Description

Original: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70: 0000012702413800 0000000064c02cd1
0x0000001b501ffb80: 0000000000000001 0000000064c021a5
0x0000001b501ffb90: 0000000000000000 0000000064c260bf
0x0000001b501ffba0: 00005593d9fd7114 0000000000000002
0x0000001b501ffbb0: 0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0: 0000000000000001 000001274a4c58d0
0x0000001b501ffbe0: 0000000000000000 0000000000000000
0x0000001b501ffbf0: 0000000000000000 0000000000000000
0x0000001b501ffc00: 0000000000000002 0000000064c32170
0x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20: 0000012702413800 0000000000000000
0x0000001b501ffc30: 0000000000000000 0000000000000000
0x0000001b501ffc40: 0000000000000000 0000000000000000
0x0000001b501ffc50: 0000000000000000 0000000000000000
0x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1ed41b]
V [jvm.dll+0x2060bf]
V [jvm.dll+0x20ac9d]
V [jvm.dll+0x212170]
V [jvm.dll+0x263c6a]
C [msvcr120.dll+0x24f7f]
C [msvcr120.dll+0x25126]
C [KERNEL32.DLL+0x17974]
C [ntdll.dll+0x6a271]

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
Begin Bits: [0x000001275d570000, 0x000001276d360000)
End Bits: [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
total_blobs=2958 nmethods=2514 adapters=354
compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

--------------- S Y S T E M ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

New: Jenkins crashes almost every hour right around 50th minute of the hour. Tried looking through many different tickets here and other online sources but couldn't really find a relevant answer.

When the Agent crashes, it produces a minidump with logs. Then it restarts itself. Another thing I noticed is that Splunk does not catch this crash event for some reason.
{noformat}
#
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_IN_PAGE_ERROR (0xc0000006) at pc=0x0000000064c0d41b, pid=1200, tid=0x000000000000267c
#
# JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x1ed41b]
#
# Failed to write core dump. Call to MiniDumpWriteDump() failed (Error 0x8007012b: Only part of a ReadProcessMemory or WriteProcessMemory request was completed.)
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

--------------- T H R E A D ---------------
Current thread (0x0000012702413800): WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]
siginfo: ExceptionCode=0xc0000006, ExceptionInformation=0x0000000000000001 0x000001274acb1420 0x00000000c0000010

Registers:
RAX=0x0000000009976d58, RBX=0x000001274a4d3170, RCX=0x000001274acb1420, RDX=0x0000000000000033
RSP=0x0000001b501ffb70, RBP=0x0000000000000002, RSI=0x000001274a4b5690, RDI=0x0000000000000000
R8=0x0000000000000031, R9 =0x00000127023e4983, R10=0x00000000546c6148, R11=0xffffffffb9e4ccfa
R12=0x0000000000000000, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x0000000064c0d41b, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x0000001b501ffb70)
0x0000001b501ffb70: 0000012702413800 0000000064c02cd1
0x0000001b501ffb80: 0000000000000001 0000000064c021a5
0x0000001b501ffb90: 0000000000000000 0000000064c260bf
0x0000001b501ffba0: 00005593d9fd7114 0000000000000002
0x0000001b501ffbb0: 0000001b501ffe10 0000000064c32926
0x0000001b501ffbc0: 00000000651fc948 0000000064c2ac9d
0x0000001b501ffbd0: 0000000000000001 000001274a4c58d0
0x0000001b501ffbe0: 0000000000000000 0000000000000000
0x0000001b501ffbf0: 0000000000000000 0000000000000000
0x0000001b501ffc00: 0000000000000002 0000000064c32170
0x0000001b501ffc10: 0000001b501ffe10 000001277fb2ab30
0x0000001b501ffc20: 0000012702413800 0000000000000000
0x0000001b501ffc30: 0000000000000000 0000000000000000
0x0000001b501ffc40: 0000000000000000 0000000000000000
0x0000001b501ffc50: 0000000000000000 0000000000000000
0x0000001b501ffc60: 000001277fb2ab30 0000000064c83c6a

Instructions: (pc=0x0000000064c0d41b)
0x0000000064c0d3fb: 5f c3 cc cc cc 40 53 48 83 ec 20 48 8b d9 48 8b
0x0000000064c0d40b: 49 38 48 85 c9 74 0c 48 8b 01 ff 10 48 8b 4b 28
0x0000000064c0d41b: 48 89 01 48 83 c4 20 5b c3 cc cc cc cc cc cc cc
0x0000000064c0d42b: cc cc cc cc cc 48 89 5c 24 08 57 48 83 ec 20 48

Register to memory mapping:
RAX=0x0000000009976d58 is an unknown value
RBX=0x000001274a4d3170 is an unknown value
RCX=0x000001274acb1420 is an unknown value
RDX=0x0000000000000033 is an unknown value
RSP=0x0000001b501ffb70 is an unknown value
RBP=0x0000000000000002 is an unknown value
RSI=0x000001274a4b5690 is an unknown value
RDI=0x0000000000000000 is an unknown value
R8 =0x0000000000000031 is an unknown value
R9 =0x00000127023e4983 is an unknown value
R10=0x00000000546c6148 is an unknown value
R11=0xffffffffb9e4ccfa is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value

Stack: [0x0000001b50100000,0x0000001b50200000], sp=0x0000001b501ffb70, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x1ed41b]
V [jvm.dll+0x2060bf]
V [jvm.dll+0x20ac9d]
V [jvm.dll+0x212170]
V [jvm.dll+0x263c6a]
C [msvcr120.dll+0x24f7f]
C [msvcr120.dll+0x25126]
C [KERNEL32.DLL+0x17974]
C [ntdll.dll+0x6a271]

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000012704b56000 JavaThread "pool-1-thread-188" daemon [_thread_blocked, id=9992, stack(0x0000001b50c00000,0x0000001b50d00000)]
0x0000012704b52000 JavaThread "pool-1-thread-187" daemon [_thread_blocked, id=5880, stack(0x0000001b50900000,0x0000001b50a00000)]
0x0000012704b4e000 JavaThread "pool-1-thread-186" daemon [_thread_blocked, id=712, stack(0x0000001b50800000,0x0000001b50900000)]
0x0000012704b51000 JavaThread "pool-1-thread-185" daemon [_thread_blocked, id=3380, stack(0x0000001b50600000,0x0000001b50700000)]
0x0000012704b52800 JavaThread "pool-1-thread-183" daemon [_thread_blocked, id=9284, stack(0x0000001b4db00000,0x0000001b4dc00000)]
0x0000012704b4f800 JavaThread "pool-1-thread-182" daemon [_thread_blocked, id=9016, stack(0x0000001b4da00000,0x0000001b4db00000)]
0x0000012704b53800 JavaThread "python D:\Temp\jenkins8775386052577470049.py: stdout copier" daemon [_thread_in_native, id=4936, stack(0x0000001b50b00000,0x0000001b50c00000)]
0x0000012704b54800 JavaThread "pool-1-thread-28 for JNLP4-connect connection to sea-an-build-06.corp.arena.net/10.25.167.30:63344 id=92" daemon [_thread_in_native, id=600, stack(0x0000001b4dc00000,0x0000001b4dd00000)]
0x0000012702353800 JavaThread "Ping thread for channel hudson.remoting.Channel@7953ba77:JNLP4-connect connection to jenkins_master/master_ip:63344" daemon [_thread_blocked, id=7772, stack(0x0000001b50e00000,0x0000001b50f00000)]
0x00000127047b3800 JavaThread "RemoteInvocationHandler [#1]" daemon [_thread_blocked, id=8764, stack(0x0000001b50a00000,0x0000001b50b00000)] 0x00000127045ea000 JavaThread "pool-1-thread-3" daemon [_thread_in_native, id=4124, stack(0x0000001b50700000,0x0000001b50800000)]
0x0000012704a26800 JavaThread "Windows IOHub Watcher for IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-2" daemon [_thread_blocked, id=5652, stack(0x0000001b50500000,0x0000001b50600000)]
0x0000012704a25000 JavaThread "IOHub#1: Selector[keys:0, gen:0] / pool-1-thread-1" daemon [_thread_in_native, id=7240, stack(0x0000001b50400000,0x0000001b50500000)]
0x0000012704a14800 JavaThread "Thread-1" [_thread_blocked, id=9556, stack(0x0000001b50300000,0x0000001b50400000)]
0x0000012702412800 JavaThread "Service Thread" daemon [_thread_blocked, id=1080, stack(0x0000001b50000000,0x0000001b50100000)]
0x0000012702358000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=1320, stack(0x0000001b4ff00000,0x0000001b50000000)]
0x0000012702356800 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=7752, stack(0x0000001b4fe00000,0x0000001b4ff00000)]
0x0000012702355800 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=2096, stack(0x0000001b4fd00000,0x0000001b4fe00000)]
0x0000012702359800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=9660, stack(0x0000001b4fc00000,0x0000001b4fd00000)]
0x000001270235a800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=7804, stack(0x0000001b4fb00000,0x0000001b4fc00000)]
0x0000012702357000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5508, stack(0x0000001b4fa00000,0x0000001b4fb00000)]
0x0000012702355000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9728, stack(0x0000001b4f900000,0x0000001b4fa00000)]
0x000001270231f800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5572, stack(0x0000001b4f800000,0x0000001b4f900000)]
0x000001270231b800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=10136, stack(0x0000001b4f700000,0x0000001b4f800000)]
0x000001274a58d000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=208, stack(0x0000001b4f600000,0x0000001b4f700000)]
0x000001274a58a000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3328, stack(0x0000001b4f500000,0x0000001b4f600000)]
0x000001270230a000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=9400, stack(0x0000001b4f400000,0x0000001b4f500000)]
0x0000012702308800 JavaThread "Attach Listener" daemon [_thread_blocked, id=9000, stack(0x0000001b4f300000,0x0000001b4f400000)]
0x000001277ff29000 JavaThread "Finalizer" daemon [_thread_blocked, id=7008, stack(0x0000001b4f200000,0x0000001b4f300000)]
0x00000127022f7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=1848, stack(0x0000001b4f100000,0x0000001b4f200000)]
0x000001274a4c7000 JavaThread "main" [_thread_blocked, id=2796, stack(0x0000001b4dd00000,0x0000001b4de00000)]

Other Threads:
0x00000127022d4000 VMThread [stack: 0x0000001b4f000000,0x0000001b4f100000] [id=10112]
=>0x0000012702413800 WatcherThread [stack: 0x0000001b50100000,0x0000001b50200000] [id=9852]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000001274a4c3a40] Threads_lock - owner thread: 0x0000012704b55000
[0x000001274a4c58d0] PeriodicTask_lock - owner thread: 0x0000012702413800

heap address: 0x00000003c8400000, size: 16252 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000

Heap:
PSYoungGen total 303616K, used 167053K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 60% used [0x000000066d700000,0x0000000677076d58,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000)
object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 18083K, capacity 18684K, committed 18816K, reserved 1067008K
class space used 2025K, capacity 2173K, committed 2176K, reserved 1048576K

Card table byte_map: [0x000001275a080000,0x000001275c040000] byte_map_base: 0x000001275823e000

Marking Bits: (ParMarkBitMap*) 0x0000000065207790
Begin Bits: [0x000001275d570000, 0x000001276d360000)
End Bits: [0x000001276d360000, 0x000001277d150000)

Polling page: 0x000001274ac90000

CodeCache: size=245760Kb used=10231Kb max_used=11931Kb free=235528Kb
bounds [0x000001274acc0000, 0x000001274b890000, 0x0000012759cc0000]
total_blobs=2958 nmethods=2514 adapters=354
compilation: enabled

Compilation events (10 events):
Event: 3581.232 Thread 0x0000012702356800 2988 ! 3 hudson.remoting.Channel$1::handle (233 bytes)
Event: 3581.235 Thread 0x0000012702355800 nmethod 2987 0x000001274b1a8950 code [0x000001274b1a8b60, 0x000001274b1a96a0]
Event: 3581.236 Thread 0x000001274a58a000 nmethod 2986 0x000001274b124a10 code [0x000001274b124b60, 0x000001274b124d88]
Event: 3581.236 Thread 0x0000012702356800 nmethod 2988 0x000001274b3bc150 code [0x000001274b3bc720, 0x000001274b3c0cd0]
Event: 3582.161 Thread 0x000001274a58d000 2989 4 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync::readerShouldBlock (5 bytes)
Event: 3582.161 Thread 0x000001274a58d000 nmethod 2989 0x000001274b3bbf10 code [0x000001274b3bc040, 0x000001274b3bc0b8]
Event: 3602.216 Thread 0x0000012702355000 2990 ! 4 java.io.ObjectStreamClass::getReflector (336 bytes)
Event: 3602.237 Thread 0x000001274a58a000 2991 4 java.util.concurrent.AbstractExecutorService::newTaskFor (9 bytes)Event: 3602.237 Thread 0x000001274a58a000 nmethod 2991 0x000001274b035a10 code [0x000001274b035b60, 0x000001274b035c78]
Event: 3602.248 Thread 0x0000012702355000 nmethod 2990 0x000001274b10f090 code [0x000001274b10f3c0, 0x000001274b111660]

GC Heap History (2 events):
Event: 242.224 GC heap before
{Heap before GC invocations=1 (full 0):
PSYoungGen total 303616K, used 260608K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000)
eden space 260608K, 100% used [0x000000066d700000,0x000000067d580000,0x000000067d580000)
from space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
to space 43008K, 0% used [0x000000067d580000,0x000000067d580000,0x000000067ff80000)
ParOldGen total 693760K, used 0K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c8400000,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
Event: 242.238 GC heap after
Heap after GC invocations=1 (full 0):
PSYoungGen total 303616K, used 9905K [0x000000066d700000, 0x0000000682980000, 0x00000007c0000000) eden space 260608K, 0% used [0x000000066d700000,0x000000066d700000,0x000000067d580000)
from space 43008K, 23% used [0x000000067d580000,0x000000067df2c6b8,0x000000067ff80000)
to space 43008K, 0% used [0x000000067ff80000,0x000000067ff80000,0x0000000682980000)
ParOldGen total 693760K, used 112K [0x00000003c8400000, 0x00000003f2980000, 0x000000066d700000) object space 693760K, 0% used [0x00000003c8400000,0x00000003c841c010,0x00000003f2980000)
Metaspace used 17478K, capacity 18006K, committed 18304K, reserved 1064960K
class space used 1965K, capacity 2091K, committed 2176K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 182.229 Thread 0x0000012704b4f800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b4c6658 method=java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$Transfe
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b5d2fc0 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.239 Thread 0x00000127045ea000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274afd3398 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 17
Event: 242.240 Thread 0x000001277ff29000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6398dc method=java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; @ 4
Event: 242.378 Thread 0x00000127047b3800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b6449c4 method=hudson.remoting.RemoteInvocationHandler$Unexporter.run()V @ 176
Event: 258.896 Thread 0x0000012704b50800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b64e2ec method=java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V @ 477
Event: 259.032 Thread 0x0000012704b4f800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x000001274b3cca6c method=java.util.regex.Pattern$Curly.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19
Event: 1562.227 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bc44 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(I)I @ 81
Event: 1562.228 Thread 0x0000012702353800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b79bfe8 method=java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(I)Z @ 9
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000001274b7a8d10 method=java.util.WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry; @ 35

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 2.234 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x000000067380a160) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.294 Thread 0x00000127047e8800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673decfd0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.766 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673b65f98) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.837 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675c181c8) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 2.850 Thread 0x0000012704b52000 Exception <a 'java/io/FileNotFoundException'> (0x0000000675c48c68) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 711]
Event: 2.894 Thread 0x0000012704b52000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000675ca4008) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]Event: 2.970 Thread 0x00000127045ea800 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000672a09000) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.312 Thread 0x00000127047ac000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000673c99de0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]
Event: 7.595 Thread 0x0000012704d0f800 Implicit null exception at 0x000001274b05f096 to 0x000001274b05f405Event: 3580.894 Thread 0x0000012704b51000 Exception <a 'java/lang/NoSuchMethodError': <clinit>> (0x0000000676bb34e0) thrown at [J:\jenkins\tmp\workspace\build\src\hotspot\src\share\vm\prims\jni.cpp, line 1616]

Events (10 events):
Event: 3581.232 Thread 0x0000012704b52000 Uncommon trap: trap_request=0xffffff65 fr.pc=0x000001274b7a8d10
Event: 3581.232 Thread 0x0000012704b52000 DEOPT PACKING pc=0x000001274b7a8d10 sp=0x0000001b509feb30
Event: 3581.232 Thread 0x0000012704b52000 DEOPT UNPACKING pc=0x000001274ad05583 sp=0x0000001b509feae0 mode 2
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.232 Executing VM operation: RevokeBias done
Event: 3581.232 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3581.234 Executing VM operation: RevokeBiasEvent: 3581.234 Executing VM operation: RevokeBias done
Event: 3582.161 Thread 0x000001274a58d000 flushing nmethod 0x000001274b198710

Dynamic libraries:
0x00007ff7ebb00000 - 0x00007ff7ebb3a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.exe
0x00007ff96bad0000 - 0x00007ff96bcbd000 C:\Windows\SYSTEM32\ntdll.dll
0x00007ff96acc0000 - 0x00007ff96ad73000 C:\Windows\System32\KERNEL32.DLL
0x00007ff9686d0000 - 0x00007ff968965000 C:\Windows\System32\KERNELBASE.dll
0x00007ff96b9f0000 - 0x00007ff96ba93000 C:\Windows\System32\ADVAPI32.dll
0x00007ff9694a0000 - 0x00007ff96953e000 C:\Windows\System32\msvcrt.dll
0x00007ff96b0b0000 - 0x00007ff96b14e000 C:\Windows\System32\sechost.dll
0x00007ff96ab90000 - 0x00007ff96acb2000 C:\Windows\System32\RPCRT4.dll
0x00007ff968f50000 - 0x00007ff9690e7000 C:\Windows\System32\USER32.dll
0x00007ff967d70000 - 0x00007ff967d90000 C:\Windows\System32\win32u.dll
0x00007ff969540000 - 0x00007ff969569000 C:\Windows\System32\GDI32.dll
0x00007ff967d90000 - 0x00007ff967f2a000 C:\Windows\System32\gdi32full.dll
0x00007ff968c00000 - 0x00007ff968ca0000 C:\Windows\System32\msvcp_win.dll
0x00007ff968970000 - 0x00007ff968a6a000 C:\Windows\System32\ucrtbase.dll
0x00007ff957030000 - 0x00007ff9572a9000 C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17763.1282_none_de75b6f253473c0f\COMCTL32.dll
0x00007ff96ad80000 - 0x00007ff96b0ac000 C:\Windows\System32\combase.dll
0x00007ff968a70000 - 0x00007ff968aee000 C:\Windows\System32\bcryptPrimitives.dll
0x00007ff93d280000 - 0x00007ff93d36f000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcr120.dll
0x00007ff944dc0000 - 0x00007ff944e66000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\msvcp120.dll
0x0000000064a20000 - 0x0000000065286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\server\jvm.dll
0x00007ff96b4a0000 - 0x00007ff96b4a8000 C:\Windows\System32\PSAPI.DLL
0x00007ff965d70000 - 0x00007ff965d94000 C:\Windows\SYSTEM32\WINMM.dll
0x00007ff95e840000 - 0x00007ff95e84a000 C:\Windows\SYSTEM32\VERSION.dll
0x00007ff965db0000 - 0x00007ff965db9000 C:\Windows\SYSTEM32\WSOCK32.dll
0x00007ff9695f0000 - 0x00007ff96965d000 C:\Windows\System32\WS2_32.dll
0x00007ff965d40000 - 0x00007ff965d6d000 C:\Windows\SYSTEM32\winmmbase.dll
0x00007ff968680000 - 0x00007ff9686ca000 C:\Windows\System32\cfgmgr32.dll
0x00007ff965af0000 - 0x00007ff965aff000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\verify.dll
0x00007ff965a50000 - 0x00007ff965a78000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\java.dll
0x00007ff954270000 - 0x00007ff954286000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\zip.dll
0x00007ff969690000 - 0x00007ff96ab88000 C:\Windows\System32\SHELL32.dll
0x00007ff968cf0000 - 0x00007ff968d98000 C:\Windows\System32\shcore.dll
0x00007ff967f30000 - 0x00007ff96867f000 C:\Windows\System32\windows.storage.dll
0x00007ff967ac0000 - 0x00007ff967ae4000 C:\Windows\System32\profapi.dll
0x00007ff967af0000 - 0x00007ff967b4d000 C:\Windows\System32\powrprof.dll
0x00007ff96b510000 - 0x00007ff96b562000 C:\Windows\System32\shlwapi.dll
0x00007ff967b50000 - 0x00007ff967b61000 C:\Windows\System32\kernel.appcore.dll
0x00007ff968ca0000 - 0x00007ff968cb7000 C:\Windows\System32\cryptsp.dll
0x00007ff953ff0000 - 0x00007ff954013000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunec.dll
0x00007ff966e40000 - 0x00007ff966e73000 C:\Windows\system32\rsaenh.dll
0x00007ff968cc0000 - 0x00007ff968ce6000 C:\Windows\System32\bcrypt.dll
0x00007ff9679f0000 - 0x00007ff967a18000 C:\Windows\SYSTEM32\USERENV.dll
0x00007ff967490000 - 0x00007ff96749c000 C:\Windows\SYSTEM32\CRYPTBASE.dll
0x00007ff953f10000 - 0x00007ff953f2a000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\net.dll
0x00007ff9672c0000 - 0x00007ff967327000 C:\Windows\system32\mswsock.dll
0x00007ff966fe0000 - 0x00007ff96701d000 C:\Windows\SYSTEM32\IPHLPAPI.DLL
0x00007ff969660000 - 0x00007ff969668000 C:\Windows\System32\NSI.dll
0x00007ff9634d0000 - 0x00007ff9634e6000 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
0x00007ff9634b0000 - 0x00007ff9634cc000 C:\Windows\SYSTEM32\dhcpcsvc.DLL
0x00007ff967020000 - 0x00007ff9670e7000 C:\Windows\SYSTEM32\DNSAPI.dll
0x00007ff953e80000 - 0x00007ff953e92000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\nio.dll
0x00007ff965ae0000 - 0x00007ff965aec000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\sunmscapi.dll
0x00007ff967b90000 - 0x00007ff967d6c000 C:\Windows\System32\CRYPT32.dll
0x00007ff967b70000 - 0x00007ff967b82000 C:\Windows\System32\MSASN1.dll
0x00007ff967590000 - 0x00007ff9675b6000 C:\Windows\SYSTEM32\ncrypt.dll
0x00007ff967550000 - 0x00007ff96758c000 C:\Windows\SYSTEM32\NTASN1.dll
0x00007ff960140000 - 0x00007ff96014a000 C:\Windows\System32\rasadhlp.dll
0x00007ff963600000 - 0x00007ff963679000 C:\Windows\System32\fwpuclnt.dll
0x00007ff939d30000 - 0x00007ff939ec5000 C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\bin\awt.dll
0x00007ff968da0000 - 0x00007ff968e64000 C:\Windows\System32\OLEAUT32.dll
0x00007ff965c20000 - 0x00007ff965cac000 C:\Windows\SYSTEM32\apphelp.dll
0x00007ff94c620000 - 0x00007ff94c662000 D:\Temp\jna--1366022967\jna4539908599793538979.dll
0x00007ff960b50000 - 0x00007ff960d3d000 C:\Windows\system32\DBGHELP.DLL
0x00007ff9606b0000 - 0x00007ff9606da000 C:\Windows\SYSTEM32\dbgcore.DLL

VM Arguments:
jvm_args: -Xrs -DSoftKillWaitSeconds=0
java_command: D:\JENKINS\agent.jar -jnlpUrl https://master_url/computer/builder/slave-agent.jnlp -secret secret_text java_class_path (initial): D:\JENKINS\agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Program Files\AdoptOpenJDK\jre-8.0.265.01-hotspot\
PATH=c:\program files\adoptopenjdk\jre-8.0.265.01-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32\wbem;C:\Windows\system32\windowspowershell\v1.0;C:\Windows\system32\openssh;c:\salt;c:\program files\python36;d:\perl\bin;C:\Program Files\Perforce\;C:\Users\BuildSrv\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\IncrediBuild;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Users\BuildSrv\AppData\Local\Microsoft\WindowsApps
USERNAME=BuildSrv
OS=Windows_NT
PROCESSOR_IDENTIFIER=AMD64 Family 23 Model 49 Stepping 0, AuthenticAMD

--------------- S Y S T E M ---------------

OS: Windows 10.0 , 64 bit Build 17763 (10.0.17763.1158)

CPU:total 24 (initial active 24) (24 cores per cpu, 1 threads per core) family 23 model 49 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, mmxext, 3dnowpref, lzcnt, sse4a, tsc, tscinvbit, tscinv, bmi1

Memory: 4k page, physical 66564500k(26330776k free), swap 76525972k(37298184k free)

vm_info: OpenJDK 64-Bit Server VM (25.265-b01) for windows-amd64 JRE (1.8.0_265-b01), built on Jul 28 2020 15:11:27 by "jenkins" with MS VC++ 12.0 (VS2013)

time: Thu Oct 29 17:55:26 2020
timezone: Pacific Daylight Time
elapsed time: 3646 seconds (0d 1h 0m 46s){noformat}

On the agent:
{noformat}
Inbound agent connected from internal_fdqn/internal_ip:64539
Inbound agent connected from internal_fdqn/internal_ip:64539
Remoting version: 4.5
This is a Windows agentAgent successfully connected and online

ERROR: Connection terminatedjava.nio.channels.ClosedChannelException
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748){noformat}
On the master:
{noformat}
2020-10-30 02:48:54.152+0000 [id=2405] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/internal_ip:63344 remote=local_fqdn/internal_ip:62976]] / Computer.threadPoolForRemoting [#1126] for localhostname terminated: java.nio.channels.ClosedChannelException
2020-10-30 02:48:55.895+0000 [id=2414] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Connection #8 failed: java.io.EOFException
2020-10-30 02:48:55.931+0000 [id=2415] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /internal_ip:63261{noformat}
It seems that something external is terminating the connection. But JVM is crashing at the same time. So I'm not quite sure where the issue lies. Could this be a Jenkins bug?

Bryant Lee made changes - 2020-10-30 08:55

Labels

New: jenkins slave windows

Bryant Lee made changes - 2020-10-30 20:37

Priority

Original: Critical [ 2 ]

New: Blocker [ 1 ]

Bryant Lee made changes - 2020-10-30 20:39

Component/s

New: remoting [ 15489 ]

Mark Waite added a comment - 2020-11-01 01:50

Hint on stackoverflow is a possible corruption of the Java installation on Windows.

https://stackoverflow.com/questions/44327267/java-exception-in-page-error

https://stackoverflow.com/questions/44327267/java-exception-in-page-error suggests it could be a hardware fault.

Mark Waite added a comment - 2020-11-01 01:50 Hint on stackoverflow is a possible corruption of the Java installation on Windows. https://stackoverflow.com/questions/44327267/java-exception-in-page-error https://stackoverflow.com/questions/44327267/java-exception-in-page-error suggests it could be a hardware fault.

Bryant Lee added a comment - 2020-11-02 18:04

markewaite

I've tried with multiple different Java and got the same results. I've tried with AdoptOpenJDK 8 and 11. I've tried with older Oracle JRE 8. OpenJDK 8. All with the same result. We are experiencing this same exact issue across 4 of our new server machines. So I doubt it's a disk issue. The only major difference between the old and new machines is the processor. Old machines have various types of Intels, whereas the new machines have AMD EPYC 7402P in them.

Bryant Lee added a comment - 2020-11-02 18:04 markewaite I've tried with multiple different Java and got the same results. I've tried with AdoptOpenJDK 8 and 11. I've tried with older Oracle JRE 8. OpenJDK 8. All with the same result. We are experiencing this same exact issue across 4 of our new server machines. So I doubt it's a disk issue. The only major difference between the old and new machines is the processor. Old machines have various types of Intels, whereas the new machines have AMD EPYC 7402P in them.

Assignee:: Unassigned

Reporter:: Bryant Lee

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2020-10-30 08:22

Updated:: 2024-08-17 17:51

Resolved:: 2024-08-17 17:51

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Mark Waite added a comment - 2020-11-01 01:50

Expand comment: Mark Waite added a comment - 2020-11-01 01:50

Collapse comment: Bryant Lee added a comment - 2020-11-02 18:04

Expand comment: Bryant Lee added a comment - 2020-11-02 18:04

People

Dates