• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • maven-plugin
    • None
    • Jenkins 1.565.1, SLES 11.3
      Reproduced Jenkins 1.565.1, maven-plugin 2.3, Java 1.7, Windows 8 (64bits)

      A "maven project" job started a Maven build which ran into an OOM/PermGen. So after 15 Minutes, the build should have been canceled by the build-timeout-plugin, but this doesn't happen. Jenkins seems to be waiting for some reaction from the zombie process instead of killing it.

      This is the end of the console output at about 1h after the job started, so 45 min after it should have been terminated:

      [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mm-content ---
      [INFO] Building jar: /Disk1/home/hudson/hudson/jobs/__jobname__/workspace/mm/distribution/dist-mm-sharedcontent/target/mm-content-9.56.0-SNAPSHOT.jar
      Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: PermGen space
      Build timed out (after 15 minutes). Marking the build as failed.
      

      The forked process it is waiting for is running at 100% CPU load, but it is no longer responding to jstack calls. jstack -F however works and shows the following:

      hudson@cihudson:~> jstack 7439
      7439: Unable to open socket file: target process not responding or HotSpot VM not loaded
      The -F option can be used when the target process is not responding
      hudson@cihudson:~> jstack -F 7439
      Attaching to process ID 7439, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 20.51-b01
      Deadlock Detection:
      
      java.lang.NullPointerException
              at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426)
              at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137)
              at sun.jvm.hotspot.oops.Oop.isA(Oop.java:100)
              at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93)
              at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39)
              at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52)
              at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
              at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
              at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
              at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
              at sun.tools.jstack.JStack.main(JStack.java:84)
      Can't print deadlocks:null
      Thread 3869: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=0, line=168 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame)
      
      
      Thread 7638: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7637: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7636: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7635: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7634: (state = BLOCKED)
       - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
       - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
       - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
       - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=957 (Interpreted frame)
       - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=23, line=917 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7630: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame)
       - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame)
      
      
      Thread 7629: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.sleep(long) @bci=52, line=188 (Interpreted frame)
       - org.eclipse.core.internal.jobs.WorkerPool.startJob(org.eclipse.core.internal.jobs.Worker) @bci=78, line=220 (Interpreted frame)
       - org.eclipse.core.internal.jobs.Worker.run() @bci=318, line=50 (Interpreted frame)
      
      
      Thread 7503: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - org.eclipse.core.internal.jobs.InternalWorker.run() @bci=135, line=58 (Interpreted frame)
      
      
      Thread 7502: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolManagerImpl.timer(int) @bci=34, line=174 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerQueueNode.run() @bci=38, line=125 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run() @bci=101, line=70 (Interpreted frame)
      
      
      Thread 7501: (state = BLOCKED)
       - org.eclipse.equinox.internal.util.impl.tpt.threadpool.ThreadPoolFactoryImpl.execute0(java.lang.Runnable, int, java.lang.String, java.security.AccessControlContext) @bci=65, line=100 (Interpreted frame)
       - org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run() @bci=313, line=110 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7497: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame)
      
      
      Thread 7496: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent() @bci=4, line=400 (Interpreted frame)
       - org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run() @bci=1, line=336 (Interpreted frame)
      
      
      Thread 7492: (state = BLOCKED)
       - org.eclipse.osgi.framework.internal.core.Framework.run() @bci=7, line=1862 (Interpreted frame)
       - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
      
      
      Thread 7450: (state = BLOCKED)
       - java.security.AccessControlContext.optimize() @bci=23, line=403 (Compiled frame; information may be imprecise)
       - java.security.AccessController.getContext() @bci=19, line=493 (Interpreted frame)
       - java.lang.Thread.init(java.lang.ThreadGroup, java.lang.Runnable, java.lang.String, long) @bci=135, line=358 (Interpreted frame)
       - java.lang.Thread.<init>(java.lang.ThreadGroup, java.lang.String) @bci=45, line=476 (Interpreted frame)
      
      
      Thread 7449: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Compiled frame)
       - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
       - java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=171 (Compiled frame)
      
      
      Thread 7448: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
       - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
      
      
      Thread 7440: (state = BLOCKED)
       - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
       - hudson.remoting.Channel.join() @bci=18, line=874 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean, hudson.remoting.JarCache) @bci=108, line=525 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode, boolean) @bci=26, line=499 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream, hudson.remoting.Channel$Mode) @bci=4, line=491 (Interpreted frame)
       - hudson.remoting.Launcher.main(java.io.InputStream, java.io.OutputStream) @bci=5, line=487 (Interpreted frame)
       - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
       - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
       - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
       - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)
       - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.io.File, java.io.File, java.io.File, java.io.File, int) @bci=346, line=136 (Interpreted frame)
       - org.jvnet.hudson.maven3.agent.Maven3Main.main(java.lang.String[]) @bci=46, line=64 (Interpreted frame)
      

      I suppose that the hudson.remoting.Channel implementation is causing the job to hang because it doesn't detect that the "remote" Maven process (running locally on master) is a zombie.

          [JENKINS-24299] OutOfMemoryError cause maven projects hang

          ikedam added a comment -

          Let me see which seems the problem you faced?

          • Build timeout was not triggered.
          • Build timeout was triggered, but the build didn't terminate.

          From your log, your case seems the latter one. Is that right?

          ikedam added a comment - Let me see which seems the problem you faced? Build timeout was not triggered. Build timeout was triggered, but the build didn't terminate. From your log, your case seems the latter one. Is that right?

          T Oberlies added a comment -

          From your log, your case seems the latter one. Is that right?

          Yes, I think so.

          T Oberlies added a comment - From your log, your case seems the latter one. Is that right? Yes, I think so.

          ikedam added a comment -

          This rather looks an issue of Jenkins core, as build-timeout only aborts build steps and cannot abort hanged threads.
          I changed the component.

          Let me know followings:

          ikedam added a comment - This rather looks an issue of Jenkins core, as build-timeout only aborts build steps and cannot abort hanged threads. I changed the component. Let me know followings: The version of your Jenkins. Do you run Jenkins on master / slave mode? https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds Can you reproduce the problem if needed?

          T Oberlies added a comment -

          I'm running Jenkins 1.565.1 without slaves.

          I haven't tried to reproduce the issue, but it should be possible by setting PermGen for the Maven build to something too low.

          T Oberlies added a comment - I'm running Jenkins 1.565.1 without slaves. I haven't tried to reproduce the issue, but it should be possible by setting PermGen for the Maven build to something too low.

          ikedam added a comment -

          I'm triyng to reproduce OutOfMemoryError in a maven project by launching Jenkins with XX:PermGen, but I still fail to reproduce that.

          Could you tell me example steps to reproduce OutOfMemoryError?

          ikedam added a comment - I'm triyng to reproduce OutOfMemoryError in a maven project by launching Jenkins with XX:PermGen, but I still fail to reproduce that. Could you tell me example steps to reproduce OutOfMemoryError?

          T Oberlies added a comment -

          The OOM/PermGen happens in the forked Maven build process. So you need to set something like -XX:MaxPermSize=32m in the MAVEN_OPTS to provoke the OOM there.

          T Oberlies added a comment - The OOM/PermGen happens in the forked Maven build process. So you need to set something like -XX:MaxPermSize=32m in the MAVEN_OPTS to provoke the OOM there.

          ikedam added a comment -

          I reproduced the problem:

          1. Install Jenkins 1.565.1
          2. Install git-plugin and restart
          3. Go to Manage Jenkins > Configure System
            • Add a valid JDK (I tested with jdk1.7.0_45)
            • Add a valid Maven (I tested with auto-installed maven 3.2.1
          4. Create a maven 2/3 project
          5. Run a build

          Result:

          • The build hangs with following output in console.
            Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: PermGen space
            
            • There are cases that log is not output till abort the build (first abort). It looks that the log is buffered.
            • The child java process still remains.
          • Aborting the build does not work for the first time.
            • The child java process still remains.
          • Aborting the build again works.

          I attached console logs and outputs:

          • Console text when the build hangs: hangingConsoleText.txt
          • Thread dump (copied from the output from /threadDump) when the build hangs: hangingConsoleText.txt
          • Thread dump when the build does not hang (launched without -XX:MaxPermSize): correctThreaddump.txt
          • Output when launched maven in console: launchedFromConsole.log
            • The process doesn't hang. (the control was returned)
            • I launched it as following:
              set MAVEN_OPTS=-XX:MaxPermSize=16m
              mvn -DforkMode=never test
              

          ikedam added a comment - I reproduced the problem: Install Jenkins 1.565.1 Install git-plugin and restart Go to Manage Jenkins > Configure System Add a valid JDK (I tested with jdk1.7.0_45) Add a valid Maven (I tested with auto-installed maven 3.2.1 Create a maven 2/3 project Add a git repository https://github.com/jenkinsci/build-timeout-plugin.git Setup Build as followings: Root POM: pom.xml Goals and options: -DforkMode=never test MAVEN_OPTS: -XX:MaxPermSize=16m Run a build Result: The build hangs with following output in console. Exception in thread "Channel reader thread: channel" java.lang.OutOfMemoryError: PermGen space There are cases that log is not output till abort the build (first abort). It looks that the log is buffered. The child java process still remains. Aborting the build does not work for the first time. The child java process still remains. Aborting the build again works. I attached console logs and outputs: Console text when the build hangs: hangingConsoleText.txt Thread dump (copied from the output from /threadDump) when the build hangs: hangingConsoleText.txt Thread dump when the build does not hang (launched without -XX:MaxPermSize): correctThreaddump.txt Output when launched maven in console: launchedFromConsole.log The process doesn't hang. (the control was returned) I launched it as following: set MAVEN_OPTS=-XX:MaxPermSize=16m mvn -DforkMode=never test

            Unassigned Unassigned
            oberlies T Oberlies
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: