[JENKINS-69921] docker plugin agents fail with I/O error and timeout exception

Type: Bug
Resolution: Unresolved
Priority: Blocker
Component/s: docker-plugin
Labels:
Environment:
Jenkins 2.356
docker-plugin 1.2.10
docker-java-api-plugin 3.2.13-37.vf3411c9828b9

Similar Issues:
Powered by SuggestiMate

Show

Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).

We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.

Build log

Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
    at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
    at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:142)
    at hudson.remoting.Command.readFrom(Command.java:128)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.Request.call(Request.java:199)
        at hudson.remoting.Channel.call(Channel.java:999)
        at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
        at hudson.model.Run.execute(Run.java:1897)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:107)
        at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
    at hudson.remoting.Request.abort(Request.java:345)
    at hudson.remoting.Channel.terminate(Channel.java:1080)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE

jenkins.log

Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54    at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)    at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)    at hudson.remoting.Command.readFrom(Command.java:142)    at hudson.remoting.Command.readFrom(Command.java:128)    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.

agent console.log

Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
    at hudson.remoting.Request$1.get(Request.java:321)
    at hudson.remoting.Request$1.get(Request.java:240)
    at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
    at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
    at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
    at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Screenshot 2022-10-21 102206.png
63 kB
2022-10-21 14:22

Pierce Jorgensen created issue - 2022-10-21 14:26

Pierce Jorgensen made changes - 2022-10-21 14:30

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName Running as SYSTEM[EnvInject] - Loading node environment variables.Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh--------------------- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ---------------------
...
--------------------- EXITING ---------------------FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:999) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:107) at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:345) at hudson.remoting.Channel.terminate(Channel.java:1080) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)Started calculate disk usage of buildFinished Calculation of disk usage of build in 0 secondsNotifying upstream projects of job completionFinished: FAILURE{code}

jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}

Agent console.log

{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.logHTTP/1.1 101 UPGRADEDContent-Type: application/vnd.docker.raw-streamConnection: UpgradeUpgrade: tcp<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55Launcher: DockerDelegatingComputerLauncherCommunication Protocol: Standard in/outThis is a Unix agentEvacuated stdoutAgent successfully connected and onlineERROR: Failed to monitor for ArchitectureERROR: Failed to monitor for Free Disk SpaceERROR: Failed to monitor for Clock DifferenceERROR: Failed to monitor for Response TimeERROR: Failed to monitor for Free Temp SpaceERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:321) at hudson.remoting.Request$1.get(Request.java:240) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
--------------------- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ---------------------

...

--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.logHTTP/1.1 101 UPGRADEDContent-Type: application/vnd.docker.raw-streamConnection: UpgradeUpgrade: tcp<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55Launcher: DockerDelegatingComputerLauncherCommunication Protocol: Standard in/outThis is a Unix agentEvacuated stdoutAgent successfully connected and onlineERROR: Failed to monitor for ArchitectureERROR: Failed to monitor for Free Disk SpaceERROR: Failed to monitor for Clock DifferenceERROR: Failed to monitor for Response TimeERROR: Failed to monitor for Free Temp SpaceERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:321) at hudson.remoting.Request$1.get(Request.java:240) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:34

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
--------------------- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ---------------------

...

--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.logHTTP/1.1 101 UPGRADEDContent-Type: application/vnd.docker.raw-streamConnection: UpgradeUpgrade: tcp<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55Launcher: DockerDelegatingComputerLauncherCommunication Protocol: Standard in/outThis is a Unix agentEvacuated stdoutAgent successfully connected and onlineERROR: Failed to monitor for ArchitectureERROR: Failed to monitor for Free Disk SpaceERROR: Failed to monitor for Clock DifferenceERROR: Failed to monitor for Response TimeERROR: Failed to monitor for Free Temp SpaceERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:321) at hudson.remoting.Request$1.get(Request.java:240) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
---- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ----
sh command output...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:35

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
---- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ----
sh command output...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
---- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ----
sh command output...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:35

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
---- AGENT cftsim002rh8dockerattach-003a9i61aqdjv ONLINE ----
sh command output...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:36

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

Build log
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
jenkins.log
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
Agent console.log
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

*Build log*
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
*jenkins.log*
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
*agent console.log*
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:42

Description

Original: Containers fail to shut down and communicate properly with Jenkins master when using the docker container attach method. The job runs fine and completes it's task of running a simple sh command, but the container can't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some of our jobs that use DDS (The SSH method unfortunately squashes a docker macvlan network providing DDS).

*Build log*
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
*jenkins.log*
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
*agent console.log*
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

New: Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).

We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.

*Build log*
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
*jenkins.log*
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
*agent console.log*
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 14:42

Description

Original: Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.

We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).

We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.

*Build log*
{code:java}
Started by user myUserName
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe
[podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh
------------------- AGENT ONLINE ------------------
Running sh command...
--------------------- EXITING ---------------------
FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54
java.io.StreamCorruptedException: invalid stream header: 6E672E54
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940)
at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:142)
at hudson.remoting.Command.readFrom(Command.java:128)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
at hudson.remoting.Request.call(Request.java:199)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1897)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:107)
at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:345)
at hudson.remoting.Channel.terminate(Channel.java:1080)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90)
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Notifying upstream projects of job completionFinished: FAILURE{code}
*jenkins.log*
{code:java}
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent
Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated
Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started
Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update
Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s)
Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.{code}
*agent console.log*
{code:java}
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
<===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55
Launcher: DockerDelegatingComputerLauncher
Communication Protocol: Standard in/out
This is a Unix agentEvacuated stdoutAgent successfully connected and online
ERROR: Failed to monitor for Architecture
ERROR: Failed to monitor for Free Disk Space
ERROR: Failed to monitor for Clock Difference
ERROR: Failed to monitor for Response Time
ERROR: Failed to monitor for Free Temp Space
ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException
at hudson.remoting.Request$1.get(Request.java:321)
at hudson.remoting.Request$1.get(Request.java:240)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112)
at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) {code}

Pierce Jorgensen made changes - 2022-10-21 15:37

Summary

Original: docker plugin fails with I/O error and timeout exception

New: docker plugin agents fail with I/O error and timeout exception

Assignee:: Pierce Jorgensen

Reporter:: Pierce Jorgensen

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2022-10-21 14:26

Updated:: 2022-10-21 15:37

Jenkins

Details

Description

Attachments

Attachments

Activity

People

Dates