-
Bug
-
Resolution: Unresolved
-
Blocker
-
Jenkins 2.356
docker-plugin 1.2.10
docker-java-api-plugin 3.2.13-37.vf3411c9828b9
Container agents fail to shut down and communicate properly with Jenkins master when using the "docker container attach" method for communication. The job runs fine and completes it's task of running a simple sh command, but the agent won't provide monitoring data and fails with a timeout exception.
We have no issues running containers using the SSH connection method, but we need the attach container method to replace JNLP for some jobs that won't work with ssh (These jobs use a docker macvlan network to share data over DDS).
We would continue to use JNLP, but it's become very unstable since it went unsupported in Java 11.
Build log
Started by user myUserName Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on cftsim002rh8dockerattach-003a9i61aqdjv on cftsim002_rh8_new_test (test-docker-attach) in workspace /jenkins/agents/work/dynamic/workspace/podman_agent_probe [podman_agent_probe] $ /bin/bash /tmp/jenkins5974481424239551146.sh ------------------- AGENT ONLINE ------------------ Running sh command... --------------------- EXITING --------------------- FATAL: java.io.StreamCorruptedException: invalid stream header: 6E672E54 java.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to cftsim002rh8dockerattach-003a9i61aqdjv at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:999) at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) at hudson.model.Run.execute(Run.java:1897) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) at hudson.model.ResourceController.execute(ResourceController.java:107) at hudson.model.Executor.run(Executor.java:449)Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:345) at hudson.remoting.Channel.terminate(Channel.java:1080) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:90) Started calculate disk usage of build Finished Calculation of disk usage of build in 0 seconds Notifying upstream projects of job completionFinished: FAILURE
jenkins.log
Started container ID c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5 for node cftsim002rh8dockerattach-003a9i61aqdjv from image: rhel8-cft:latest Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): [conmon:d]: exec with attach is waiting for start message from parent[conmon:d]: exec with attach got start message from parent Oct 21, 2022 9:48:26 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): Oct 21, 2022 1:48:26 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Using /jenkins/agents/work/dynamic/agent.log as an agent error log destination; output log will not be generated Oct 21, 2022 9:48:27 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal stderr from cftsim002rh8dockerattach-003a9i61aqdjv (c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5): channel started Oct 21, 2022 9:48:35 AM INFO hudson.slaves.NodeProvisioner update Image of rhel8-cft:latest provisioning successfully completed. We have now 47 computer(s) Oct 21, 2022 9:49:56 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run I/O error in channel cftsim002rh8dockerattach-003a9i61aqdjvjava.io.StreamCorruptedException: invalid stream header: 6E672E54 at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:940) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println Disconnected computer for node 'cftsim002rh8dockerattach-003a9i61aqdjv'. Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println Removed Node for node 'cftsim002rh8dockerattach-003a9i61aqdjv'. Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println Stopped container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'. Oct 21, 2022 9:49:56 AM INFO io.jenkins.docker.DockerTransientNode$1 println Removed container 'c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5' for node 'cftsim002rh8dockerattach-003a9i61aqdjv'.
agent console.log
Connecting to docker container c8a1d888a70ffa56e0a6aca905dc9604ab90bf31fb45b511e285d95ff021def5, running command java -Xmx2048m -jar /jenkins/agents/work/dynamic/agent.jar -noReconnect -noKeepAlive -agentLog /jenkins/agents/work/dynamic/agent.log HTTP/1.1 101 UPGRADED Content-Type: application/vnd.docker.raw-stream Connection: Upgrade Upgrade: tcp <===[JENKINS REMOTING CAPACITY]===>Remoting version: 3025.vf64a_a_3da_6b_55 Launcher: DockerDelegatingComputerLauncher Communication Protocol: Standard in/out This is a Unix agentEvacuated stdoutAgent successfully connected and online ERROR: Failed to monitor for Architecture ERROR: Failed to monitor for Free Disk Space ERROR: Failed to monitor for Clock Difference ERROR: Failed to monitor for Response Time ERROR: Failed to monitor for Free Temp Space ERROR: Failed to monitor for Free Swap Spacejava.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:321) at hudson.remoting.Request$1.get(Request.java:240) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:112) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
