-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
-
GSoC - Coding Phase 3
Remoting Kafka Agent classloading breaks in some cases:
1) Remoting class fetching is asynchronous operation in the Channel instance. If something fails, classloading for the class will not be retried until the channel is reset
2) Remoting Kafka Agent holds the connection Channel, so it will stay waiting for classloading forever, even if the master receives command and then fails to respond
3) One of the response failures I see in debug (Classloader fetch3 command was received, but the response didn't get sent back):
INFO: [Consumer clientId=consumer-1, groupId=hello] Setting newly assigned partitions [test2-localhost-8080-connect-0] Jun 10, 2018 1:53:38 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 1:53:45 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 1:53:55 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator markCoordinatorUnknown INFO: [Consumer clientId=consumer-1, groupId=hello] Group coordinator 127.0.0.1:9092 (id: 2147482646 rack: null) is unavailable or invalid, will attempt rediscovery Jun 10, 2018 1:53:55 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler onSuccess INFO: [Consumer clientId=consumer-1, groupId=hello] Discovered group coordinator 127.0.0.1:9092 (id: 2147482646 rack: null) Jun 10, 2018 1:53:55 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator markCoordinatorUnknown INFO: [Consumer clientId=consumer-1, groupId=hello] Group coordinator 127.0.0.1:9092 (id: 2147482646 rack: null) is unavailable or invalid, will attempt rediscovery Jun 10, 2018 1:53:55 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 1:53:56 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler onSuccess INFO: [Consumer clientId=consumer-1, groupId=hello] Discovered group coordinator 127.0.0.1:9092 (id: 2147482646 rack: null) Jun 10, 2018 1:53:56 PM org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler handle SEVERE: [Consumer clientId=consumer-1, groupId=hello] Offset commit failed on partition test2-localhost-8080-connect-0 at offset 31: The coordinator is not aware of this member. Jun 10, 2018 1:53:56 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: Unexpected error in channel test2 org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:775) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:726) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:822) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:802) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:563) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:390) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:209) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:597) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1218) at io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport.read(KafkaClassicCommandTransport.java:84) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Jun 10, 2018 1:53:56 PM hudson.remoting.SynchronousCommandTransport$ReaderThread lambda$new$0 SEVERE: Uncaught exception in SynchronousCommandTransport.ReaderThread Thread[Channel reader thread: test2,5,main] org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:713) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:596) at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1218) at io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport.closeRead(KafkaClassicCommandTransport.java:67) at hudson.remoting.Channel.terminate(Channel.java:1031) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:99)
[JENKINS-51842] Unreliable classloading chain
Without Debugger it's getting pretty far:
Master:
Jun 10, 2018 2:00:28 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo <init> INFO: Kafka version : 1.1.0 Jun 10, 2018 2:00:28 PM org.apache.kafka.common.utils.AppInfoParser$AppInfo <init> INFO: Kafka commitId : fdcf75ea326b8e07 Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.Metadata update INFO: Cluster ID: nEbZ2eilTDCRqsqLyfC5wQ Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.Metadata update INFO: Cluster ID: nEbZ2eilTDCRqsqLyfC5wQ Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler onSuccess INFO: [Consumer clientId=consumer-1, groupId=hello] Discovered group coordinator 127.0.0.1:9092 (id: 2147482646 rack: null) Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.consumer.internals.ConsumerCoordinator onJoinPrepare INFO: [Consumer clientId=consumer-1, groupId=hello] Revoking previously assigned partitions [] Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator sendJoinGroupRequest INFO: [Consumer clientId=consumer-1, groupId=hello] (Re-)joining group Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@1064f984, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1 onSuccess INFO: [Consumer clientId=consumer-1, groupId=hello] Successfully joined group with generation 65 Jun 10, 2018 2:00:28 PM org.apache.kafka.clients.consumer.internals.ConsumerCoordinator onJoinComplete INFO: [Consumer clientId=consumer-1, groupId=hello] Setting newly assigned partitions [test2-localhost-8080-connect-0] Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.slaves.SlaveComputer$DetectOS@5168121a, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.slaves.SlaveComputer$DetectDefaultCharset@5273249b, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.slaves.SlaveComputer$SlaveInitializer@3c2db7d4, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.slaves.ChannelPinger$SetUpRemotePing@2905, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2)(java.util.HashMap), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.remoting.PingThread$Ping@54b6cb1, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@492b07bf, in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.remoting.PingThread$Ping@584e9596 Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.remoting.PingThread$Ping@584e9596(hudson.remoting.UserRequest$NormalResponse), in topic=localhost-8080-test2-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport Jun 10, 2018 2:00:31 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=localhost-8080-test2-connect, with key=launch
Agent:
Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@1064f984 [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - Cluster ID: nEbZ2eilTDCRqsqLyfC5wQ Jun 10, 2018 2:00:28 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@1064f984(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.slaves.SlaveComputer$DetectOS@5168121a Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.slaves.SlaveComputer$DetectOS@5168121a(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.slaves.SlaveComputer$DetectDefaultCharset@5273249b Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.slaves.SlaveComputer$DetectDefaultCharset@5273249b(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.slaves.SlaveComputer$SlaveInitializer@3c2db7d4 Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.slaves.SlaveComputer$SlaveInitializer@3c2db7d4(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.slaves.ChannelPinger$SetUpRemotePing@2905 Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=RPCRequest:hudson.remoting.RemoteClassLoader$IClassLoader.fetch3[java.lang.String](2), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(java.util.HashMap) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.slaves.ChannelPinger$SetUpRemotePing@2905(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:hudson.remoting.PingThread$Ping@584e9596, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: UserRequest:hudson.remoting.PingThread$Ping@54b6cb1 Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Response:UserRequest:hudson.remoting.PingThread$Ping@54b6cb1(hudson.remoting.UserRequest$NormalResponse), in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Response(hudson.remoting.UserRequest$NormalResponse) Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:30 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=Unexport, in topic=test2-localhost-8080-connect, with key=launch Jun 10, 2018 2:00:31 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport read INFO: Received a command: Unexport
W.r.t the " Successful log", we are getting an active channel which even responds to channel pinger.
But... There is hanging in setChannel() which prevents ComputerLauncher from existing:
"KafkaComputerLauncher.launch for 'test2' node [#1] / waiting for test2 id=6" #62 prio=5 os_prio=31 tid=0x00007fb8596d7800 nid=0xf607 in Object.wait() [0x000070000c1ff000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:177) - locked <0x00000006c2dfb3c0> (a hudson.remoting.UserRequest) at hudson.remoting.Channel.call(Channel.java:954) at jenkins.slaves.StandardOutputSwapper.preOnline(StandardOutputSwapper.java:32) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:648) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:457) at io.jenkins.plugins.remotingkafka.KafkaComputerLauncher$1.call(KafkaComputerLauncher.java:55) at io.jenkins.plugins.remotingkafka.KafkaComputerLauncher$1.call(KafkaComputerLauncher.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
This command has been sent by master but never received by the agent:
Jun 10, 2018 2:00:29 PM io.jenkins.plugins.remotingkafka.commandtransport.KafkaClassicCommandTransport write INFO: Sent a command=UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@492b07bf, in topic=localhost-8080-test2-connect, with key=launch
My proposal would be to somehow invalidate classloader upon receiving a connection request from the master side