-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 1.565.1, SLES 11.3
Reproduced Jenkins 1.565.1, maven-plugin 2.3, Java 1.7, Windows 8 (64bits)
A "maven project" job started a Maven build which ran into an OOM/PermGen. So after 15 Minutes, the build should have been canceled by the build-timeout-plugin, but this doesn't happen. Jenkins seems to be waiting for some reaction from the zombie process instead of killing it.
This is the end of the console output at about 1h after the job started, so 45 min after it should have been terminated:
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mm-content --- [INFO] Building jar: /Disk1/home/hudson/hudson/jobs/__jobname__/workspace/mm/distribution/dist-mm-sharedcontent/target/mm-content-9.56.0-SNAPSHOT.jar Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: PermGen space Build timed out (after 15 minutes). Marking the build as failed.
The forked process it is waiting for is running at 100% CPU load, but it is no longer responding to jstack calls. jstack -F however works and shows the following:
hudson@cihudson:~> jstack 7439 7439: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding hudson@cihudson:~> jstack -F 7439 Attaching to process ID 7439, please wait... Debugger attached successfully. Server compiler detected. JVM version is 20.51-b01 Deadlock Detection: java.lang.NullPointerException at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426) at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137) at sun.jvm.hotspot.oops.Oop.isA(Oop.java:100) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) Can't print deadlocks:null Thread 3869: (state = BLOCKED) - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=0, line=168 (Interpreted frame) - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame) - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame) Thread 7638: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7637: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7636: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7635: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7634: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7630: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame) - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame) - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame) Thread 7629: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame) - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame) - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame) Thread 7503: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - org.eclipse.core.internal.jobs.InternalWorker.run() @bci=135, line=58 (Interpreted frame) Thread 7502: (state = BLOCKED) - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=34, line=174 (Interpreted frame) - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame) - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame) Thread 7501: (state = BLOCKED) - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolFactoryImpl.execute0(java.lang.Runnable, int, java.lang.String, java.security.AccessControlContext) @bci=65, line=100 (Interpreted frame) - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run() @bci=313, line=110 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7497: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame) - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame) Thread 7496: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame) - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame) Thread 7492: (state = BLOCKED) - org.eclipse.osgi.framework.internal.core.Framework.run() @bci=7, line=1862 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) Thread 7450: (state = BLOCKED) - java.security.AccessControlContext.optimize() @bci=23, line=403 (Compiled frame; information may be imprecise) - java.security.AccessController.getContext() @bci=19, line=493 (Interpreted frame) - java.lang.Thread.init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long) @bci=135, line=358 (Interpreted frame) - java.lang.Thread.<init>(java.lang.ThreadGroup, java.lang.String) @bci=45, line=476 (Interpreted frame) Thread 7449: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Compiled frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=171 (Compiled frame) Thread 7448: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame) Thread 7440: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - hudson.remoting.Channel.join() @bci=18, line=874 (Interpreted frame) - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean, hudson.remoting.JarCache) @bci=108, line=525 (Interpreted frame) - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean) @bci=26, line=499 (Interpreted frame) - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode) @bci=4, line=491 (Interpreted frame) - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream) @bci=5, line=487 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame) - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame) - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame) - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.io.File, java.io.File, java.io.File, java.io.File, int) @bci=346, line=136 (Interpreted frame) - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.lang.String[]) @bci=46, line=64 (Interpreted frame)
I suppose that the hudson.remoting.Channel implementation is causing the job to hang because it doesn't detect that the "remote" Maven process (running locally on master) is a zombie.
Let me see which seems the problem you faced?
From your log, your case seems the latter one. Is that right?