[JENKINS-18011] Deadlock issue remote slaves

Type: Bug
Resolution: Not A Defect
Priority: Critical
Component/s: core
Labels:
- core
- remote
- slave
Environment:
Linux Master runing programs on Windows slaves
Verified in Jenkins 1.480.1 and Jenkins 1.509.1

Similar Issues:
Powered by SuggestiMate

Show

Jenkins slaves running on Windows have a tendency to deadlock when programs are sending a lot of data simultaneously.

Reading the threaddump from VisualVM we can see that proc.java tries to
write to deadlock stream resulting in hanging builds

String msg = "Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information";
Throwable e = new Exception().fillInStackTrace();
LOGGER.log(Level.WARNING,msg,e);

"pool-1-thread-833" - Thread t@1926010
   java.lang.Thread.State: BLOCKED
	at java.util.logging.StreamHandler.publish(Unknown Source)
	- waiting to lock <1502546> (a java.util.logging.ConsoleHandler) owned by "pool-1-thread-927" t@1926604
	at java.util.logging.ConsoleHandler.publish(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at java.util.logging.Logger.doLog(Unknown Source)
	at java.util.logging.Logger.log(Unknown Source)
	at hudson.Proc$LocalProc.join(Proc.java:330)
	at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:993)
	at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:275)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:256)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:58)
	at java.lang.Thread.run(Unknown Source)

is related to

JENKINS-13330 Jenkins slave hangs in post build phase

Open

Marcus Jacobsson created issue - 2013-05-20 12:03

Marcus Jacobsson made changes - 2013-05-20 13:19

Labels

Original: slave

New: core remote slave

Marcus Jacobsson made changes - 2013-05-21 10:50

Description

Original: Jenkins slaves running on Windows have a tendency to deadlock when programs are sending a lot of data simultaneously.

Reading the threaddump from VisualVM we can see that proc.java tries to
write to deadlock stream resulting in hanging builds
"""
String msg = "Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information";
Throwable e = new Exception().fillInStackTrace();
LOGGER.log(Level.WARNING,msg,e);
"""

"pool-1-thread-833" - Thread t@1926010
java.lang.Thread.State: BLOCKED
at java.util.logging.StreamHandler.publish(Unknown Source)
- waiting to lock <1502546> (a java.util.logging.ConsoleHandler) owned by "pool-1-thread-927" t@1926604
at java.util.logging.ConsoleHandler.publish(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at hudson.Proc$LocalProc.join(Proc.java:330)
at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:993)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:275)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:256)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:58)
at java.lang.Thread.run(Unknown Source)

New: Jenkins slaves running on Windows have a tendency to deadlock when programs are sending a lot of data simultaneously.

Reading the threaddump from VisualVM we can see that proc.java tries to
write to deadlock stream resulting in hanging builds
"""
String msg = "Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information";
Throwable e = new Exception().fillInStackTrace();
LOGGER.log(Level.WARNING,msg,e);
"""
{code}
"pool-1-thread-833" - Thread t@1926010
java.lang.Thread.State: BLOCKED
at java.util.logging.StreamHandler.publish(Unknown Source)
- waiting to lock <1502546> (a java.util.logging.ConsoleHandler) owned by "pool-1-thread-927" t@1926604
at java.util.logging.ConsoleHandler.publish(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at hudson.Proc$LocalProc.join(Proc.java:330)
at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:993)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:275)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:256)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:58)
at java.lang.Thread.run(Unknown Source)
{code}

Marcus Jacobsson made changes - 2013-05-21 10:51

Description

Original: Jenkins slaves running on Windows have a tendency to deadlock when programs are sending a lot of data simultaneously.

Reading the threaddump from VisualVM we can see that proc.java tries to
write to deadlock stream resulting in hanging builds
"""
String msg = "Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information";
Throwable e = new Exception().fillInStackTrace();
LOGGER.log(Level.WARNING,msg,e);
"""
{code}
"pool-1-thread-833" - Thread t@1926010
java.lang.Thread.State: BLOCKED
at java.util.logging.StreamHandler.publish(Unknown Source)
- waiting to lock <1502546> (a java.util.logging.ConsoleHandler) owned by "pool-1-thread-927" t@1926604
at java.util.logging.ConsoleHandler.publish(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at hudson.Proc$LocalProc.join(Proc.java:330)
at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:993)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:275)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:256)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:58)
at java.lang.Thread.run(Unknown Source)
{code}

New: Jenkins slaves running on Windows have a tendency to deadlock when programs are sending a lot of data simultaneously.

Reading the threaddump from VisualVM we can see that proc.java tries to
write to deadlock stream resulting in hanging builds
{code}
String msg = "Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information";
Throwable e = new Exception().fillInStackTrace();
LOGGER.log(Level.WARNING,msg,e);
{code}
{code}
"pool-1-thread-833" - Thread t@1926010
java.lang.Thread.State: BLOCKED
at java.util.logging.StreamHandler.publish(Unknown Source)
- waiting to lock <1502546> (a java.util.logging.ConsoleHandler) owned by "pool-1-thread-927" t@1926604
at java.util.logging.ConsoleHandler.publish(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at hudson.Proc$LocalProc.join(Proc.java:330)
at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:993)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:275)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:256)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:215)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:58)
at java.lang.Thread.run(Unknown Source)
{code}

Marcus Jacobsson made changes - 2013-05-22 11:43

Priority

Original: Major [ 3 ]

New: Critical [ 2 ]

Marcus Jacobsson made changes - 2013-06-05 14:41

Resolution		New: Not A Defect [ 7 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Jesse Glick made changes - 2013-07-12 11:38

Link

New: This issue is related to JENKINS-13330 [ JENKINS-13330 ]

R. Tyler Croy made changes - 2016-07-25 23:55

Workflow

Original: JNJira [ 149308 ]

New: JNJira + In-Review [ 193063 ]

Assignee:: Unassigned

Reporter:: Marcus Jacobsson

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2013-05-20 12:03

Updated:: 2013-07-12 11:38

Resolved:: 2013-06-05 14:41

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates