Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66526

Deadlocks at remoting library 4.8

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: remoting
    • Labels:
      None
    • Environment:
      4.8
    • Similar Issues:

      Description

      Found one Java-level deadlock: ============================= "Running CpsFlowExecution[Owner[DataFactory/FAUST/Regress_Developers/Regress_Test/5651:DataFactory/FAUST/Regress_Developers/Regress_Test #5651]]": waiting to lock monitor 0x00007f14b01f5968 (object 0x0000000415327360, a java.io.PrintStream), which is held by "org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#6444]: checking /home/jenkins/agent/workspace/DataFactory/FAUST/Regress_Developers/Regress_Test/stork_dir on massive-8g-os4-three-jcp0d" "org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#6444]: checking /home/jenkins/agent/workspace/DataFactory/FAUST/Regress_Developers/Regress_Test/stork_dir on massive-8g-os4-three-jcp0d": waiting to lock monitor 0x00007f1688f7a728 (object 0x00000003c3584b18, a hudson.remoting.Channel), which is held by "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.127.12.99:8188 remote=pvlas-jenci0013.ca.sbrf.ru/10.127.12.117:35610]] / Computer.threadPoolForRemoting [#15240]" "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.127.12.99:8188 remote=pvlas-jenci0013.ca.sbrf.ru/10.127.12.117:35610]] / Computer.threadPoolForRemoting [#15240]": waiting to lock monitor 0x00007f15dc5450a8 (object 0x000000039a0dfae8, a hudson.remoting.ProxyOutputStream), which is held by "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#1957]" "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#1957]": waiting to lock monitor 0x00007f1688f7a728 (object 0x00000003c3584b18, a hudson.remoting.Channel), which is held by "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.127.12.99:8188 remote=pvlas-jenci0013.ca.sbrf.ru/10.127.12.117:35610]] / Computer.threadPoolForRemoting [#15240]"Java stack information for the threads listed above: =================================================== "Running CpsFlowExecution[Owner[DataFactory/FAUST/Regress_Developers/Regress_Test/5651:DataFactory/FAUST/Regress_Developers/Regress_Test #5651]]": at java.io.PrintStream.println(PrintStream.java:805) - waiting to lock <0x0000000415327360> (a java.io.PrintStream) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.stop(DurableTaskStep.java:514) at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:319) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$2.onSuccess(CpsBodyExecution.java:271) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$2.onSuccess(CpsBodyExecution.java:258) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:917) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#6444]: checking /home/jenkins/agent/workspace/DataFactory/FAUST/Regress_Developers/Regress_Test/stork_dir on massive-8g-os4-three-jcp0d": at hudson.remoting.Channel.send(Channel.java:761) - waiting to lock <0x00000003c3584b18> (a hudson.remoting.Channel) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:145) - locked <0x000000039a0909e8> (a hudson.remoting.ProxyOutputStream) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112) at org.apache.commons.io.output.TeeOutputStream.write(TeeOutputStream.java:62) - locked <0x0000000312c85830> (a org.apache.commons.io.output.TeeOutputStream) at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Filter$1.eol(BindingStep.java:221) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at java.io.PrintStream.write(PrintStream.java:449) - locked <0x0000000312c85868> (a java.io.PrintStream) at java.io.FilterOutputStream.write(FilterOutputStream.java:77) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$NewlineSafeTaskListener$1.write(DurableTaskStep.java:451) at java.io.FilterOutputStream.write(FilterOutputStream.java:125) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution$NewlineSafeTaskListener$1.write(DurableTaskStep.java:455) at java.io.PrintStream.write(PrintStream.java:480) - locked <0x0000000415327360> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104) - locked <0x0000000312c878b0> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185) at java.io.PrintStream.newLine(PrintStream.java:546) - locked <0x0000000415327360> (a java.io.PrintStream) at java.io.PrintStream.println(PrintStream.java:807) - locked <0x0000000415327360> (a java.io.PrintStream) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspaceProblem(DurableTaskStep.java:398) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.getWorkspace(DurableTaskStep.java:382) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:565) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:548) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IOHub#1: Worker[channel:java.nio.channels.SocketChannel[connected local=/10.127.12.99:8188 remote=pvlas-jenci0013.ca.sbrf.ru/10.127.12.117:35610]] / Computer.threadPoolForRemoting [#15240]": at hudson.remoting.ProxyOutputStream.error(ProxyOutputStream.java:169) - waiting to lock <0x000000039a0dfae8> (a hudson.remoting.ProxyOutputStream) at hudson.remoting.ExportTable.abort(ExportTable.java:400) at hudson.remoting.Channel.terminate(Channel.java:1095) - locked <0x00000003c3584b18> (a hudson.remoting.Channel) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:229) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:817) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:288) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:170) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:817) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:179) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:792) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#1957]": at hudson.remoting.Channel.send(Channel.java:761) - waiting to lock <0x00000003c3584b18> (a hudson.remoting.Channel) at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:145) - locked <0x000000039a0dfae8> (a hudson.remoting.ProxyOutputStream) at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112) at org.apache.commons.io.output.TeeOutputStream.write(TeeOutputStream.java:62) - locked <0x0000000312c1ee78> (a org.apache.commons.io.output.TeeOutputStream) at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Filter$1.eol(BindingStep.java:221) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74) at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Filter$1.eol(BindingStep.java:221) at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56) at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74) at java.io.PrintStream.write(PrintStream.java:480) - locked <0x0000000312c20f10> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104) - locked <0x0000000312c20f30> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185) at java.io.PrintStream.newLine(PrintStream.java:546) - locked <0x0000000312c20f10> (a java.io.PrintStream) at java.io.PrintStream.println(PrintStream.java:807) - locked <0x0000000312c20f10> (a java.io.PrintStream) at hudson.model.TaskListener._error(TaskListener.java:91) at hudson.model.TaskListener.fatalError(TaskListener.java:145) at org.jenkinsci.plugins.ansible.AnsiblePlaybookBuilder.perform(AnsiblePlaybookBuilder.java:266) at org.jenkinsci.plugins.ansible.workflow.AnsiblePlaybookStep$AnsiblePlaybookExecution.run(AnsiblePlaybookStep.java:430) at org.jenkinsci.plugins.ansible.workflow.AnsiblePlaybookStep$AnsiblePlaybookExecution.run(AnsiblePlaybookStep.java:351) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate2(ACL.java:449) at hudson.security.ACL.impersonate(ACL.java:461) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Found 2 deadlocks.
      

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            andreyhr Andrey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: