-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins Instance (2.135) with 5 agents all connected using Kafka (no security)
remoting Kafka plugin 1.1, built with remoting 3.23
Using Remoting Kafka Plugin to version 1.1 (changed the pom file to use the remoting.version=3.23 to match the version on my set up)
I've set a Jenkins instance with 5 nodes all connected with kafka. It did run jobs for about 3 days with a considerable load without failures. But today I had the following failure (full log attached):
Error when executing always post condition:hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on daybreakk failed. The channel is closing down or has closed down
The process that I started in the kafka agent with:
> java -jar remoting-kafka-agent.jar -name daybreakk -master http://jenkins:8081/ -secret 6e6c09c25550bd721cef2efba02bfbd54d6306fc711c2482450abb5ee454cd7f -kafkaURL kafka.al.com.au:9093 -noauth
exited without any log.
When I restarted the process, after all the kafka initial logs it printed: (full log attached)
Sep 04, 2018 8:44:47 AM hudson.remoting.ExportTable unexportByOidSEVERE: Trying to unexport an object that's already unexportedjava.util.concurrent.ExecutionException: Invalid object ID 750 iota=2 at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:516) at hudson.remoting.Channel.unexport(Channel.java:803) at hudson.remoting.Channel.unexport(Channel.java:793) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:49) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:87) Sep 04, 2018 8:44:47 AM hudson.remoting.ExportTable unexportByOidSEVERE: 2nd unexport attempt is hereCommand Unexport created at at hudson.remoting.Command.<init>(Command.java:79) at hudson.remoting.Command.<init>(Command.java:62) at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:35) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:393) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$1000(RemoteInvocationHandler.java:352) at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:610) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@2ee was created for interface hudson.plugins.git.IGitAPI at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:142) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:161) at hudson.remoting.Channel.export(Channel.java:768) at hudson.remoting.Channel.export(Channel.java:731) at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:198) at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1218) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.UserRequest._serialize(UserRequest.java:264) at hudson.remoting.UserRequest.serialize(UserRequest.java:273) at hudson.remoting.UserRequest.perform(UserRequest.java:223) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.jenkins.plugins.remotingkafka.Engine$1.lambda$newThread$0(Engine.java:47) at java.lang.Thread.run(Thread.java:748) ... ... ... Sep 04, 2018 8:44:47 AM hudson.remoting.Request$2 runINFO: Failed to send back a reply to the request hudson.remoting.Request$2@7da8bc0d: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@51a38854:daybreakk": channel is already closed[Thread-2] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 1.1.0[Thread-2] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : fdcf75ea326b8e07
it printed this repeatedly and it successfully connected at some point.
There were no meaningful logs in the support folder of the Remote root directory (full logs attached)
My expectation was that even if we had an exception like this, the job will not fail due to this cause, and keep waiting until the node gets reconnected and in the worse case, fail due to a timeout.
[JENKINS-53397] ChannelClosedException when using kafka nodes
| Attachment | New: AGENT_LOG_after_channel_closed_down.log [ 43991 ] | |
| Attachment | New: AGENT_LOG_initial.log [ 43992 ] | |
| Attachment | New: all_2018-08-31_06.00.21.log [ 43993 ] | |
| Attachment | New: all_2018-09-03_22.59.06.log [ 43994 ] | |
| Attachment | New: Jenkins_job.log [ 43995 ] |
The unexport pattern looks similar to https://issues.jenkins-ci.org/browse/JENKINS-42533?focusedCommentId=294094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-294094 . It should be fixed by upgrading Remoting to 3.26. CC jthompson just in case.
pvtuan10 my recommendation would be to update Remoting and spin another release. I am not sure it fixes the root cause of this issue, but at least it will prevent known issues.
I'll try building the kafka plugin against remoting 3.26 and updating Jenkins to 2.141
Thanks for reporting fnaum! As oleg_nenashev suggested, I will try to update Jenkins and remoting version for the plugin first to see if you are still facing any similar issue.
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
| Epic Link | New: JENKINS-53417 [ 193519 ] |
Jeff Thompson
added a comment - As Oleg mentions, upgrading to Remoting 3.26 will probably clear up that log message, "Trying to unexport an object that's already unexported". That's a good first step. There are still probably other issues after that.
Jeff Thompson
added a comment - As Oleg mentions, upgrading to Remoting 3.26 will probably clear up that log message, "Trying to unexport an object that's already unexported". That's a good first step. There are still probably other issues after that. yes, merged the PR https://github.com/jenkinsci/remoting-kafka-plugin/pull/43, I will try to release soon by the end of this week.
having problems uploading the logs. Will upload them soon