• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • maven-plugin
    • None
    • Jenkins 1.565.1, SLES 11.3
      Reproduced Jenkins 1.565.1, maven-plugin 2.3, Java 1.7, Windows 8 (64bits)

      A "maven project" job started a Maven build which ran into an OOM/PermGen. So after 15 Minutes, the build should have been canceled by the build-timeout-plugin, but this doesn't happen. Jenkins seems to be waiting for some reaction from the zombie process instead of killing it.

      This is the end of the console output at about 1h after the job started, so 45 min after it should have been terminated:

      [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mm-content ---
      [INFO] Building jar: /Disk1/home/hudson/hudson/jobs/__jobname__/workspace/mm/distribution/dist-mm-sharedcontent/target/mm-content-9.56.0-SNAPSHOT.jar
      Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: PermGen space
      Build timed out (after 15 minutes). Marking the build as failed.
      

      The forked process it is waiting for is running at 100% CPU load, but it is no longer responding to jstack calls. jstack -F however works and shows the following:

      hudson@cihudson:~> jstack 7439
      7439: Unable to open socket file: target process not responding or HotSpot VM not loaded
      The -F option can be used when the target process is not responding
      hudson@cihudson:~> jstack -F 7439
      Attaching to process ID 7439, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 20.51-b01
      Deadlock Detection:
      
      java.lang.NullPointerException
              at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426)
              at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137)
              at sun.jvm.hotspot.oops.Oop.isA(Oop.java:100)
              at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93)
              at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
              at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
              at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
              at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
              at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
              at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
              at sun.tools.jstack.JStack.main(JStack.java:84)
      Can't print deadlocks:null
      Thread 3869: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=0, line=168 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame)
      
      
      Thread 7638: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7637: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7636: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7635: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7634: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7630: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame)
       - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame)
      
      
      Thread 7629: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame)
       - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame)
      
      
      Thread 7503: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.InternalWorker.run() @bci=135, line=58 (Interpreted frame)
      
      
      Thread 7502: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=34, line=174 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame)
      
      
      Thread 7501: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolFactoryImpl.execute0(java.lang.Runnable, int, java.lang.String, java.security.AccessControlContext) @bci=65, line=100 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run() @bci=313, line=110 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7497: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame)
      
      
      Thread 7496: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame)
      
      
      Thread 7492: (state = BLOCKED)
       - org.eclipse.osgi.framework.internal.core.Framework.run() @bci=7, line=1862 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7450: (state = BLOCKED)
       - java.security.AccessControlContext.optimize() @bci=23, line=403 (Compiled frame; information may be imprecise)
       - java.security.AccessController.getContext() @bci=19, line=493 (Interpreted frame)
       - java.lang.Thread.init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long) @bci=135, line=358 (Interpreted frame)
       - java.lang.Thread.<init>(java.lang.ThreadGroup, java.lang.String) @bci=45, line=476 (Interpreted frame)
      
      
      Thread 7449: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Compiled frame)
       - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
       - java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=171 (Compiled frame)
      
      
      Thread 7448: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
       - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
      
      
      Thread 7440: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - hudson.remoting.Channel.join() @bci=18, line=874 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean, hudson.remoting.JarCache) @bci=108, line=525 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean) @bci=26, line=499 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode) @bci=4, line=491 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream) @bci=5, line=487 (Interpreted frame)
       - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
       - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)
       - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.io.File, java.io.File, java.io.File, java.io.File, int) @bci=346, line=136 (Interpreted frame)
       - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.lang.String[]) @bci=46, line=64 (Interpreted frame)
      

      I suppose that the hudson.remoting.Channel implementation is causing the job to hang because it doesn't detect that the "remote" Maven process (running locally on master) is a zombie.

          [JENKINS-24299] OutOfMemoryError cause maven projects hang

            Unassigned Unassigned
            oberlies T Oberlies
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: