-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins 2.89.4, stock Ubuntu 16.04 Xenial
We intermittently see nodes fail during Git checkout with a traceback that looks like this:
*09:45:25* java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer *09:45:25* at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29) *09:45:25* at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71) *09:45:25* at jdk.internal.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) *09:45:25* at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) *09:45:25* at java.lang.reflect.Method.invoke(Method.java:564) *09:45:25* at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:922) *09:45:25* at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:896) *09:45:25* at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853) *09:45:25* at hudson.remoting.UserRequest.perform(UserRequest.java:207) *09:45:25* at hudson.remoting.UserRequest.perform(UserRequest.java:53) *09:45:25* at hudson.remoting.Request$2.run(Request.java:358) *09:45:25* at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) *09:45:25* at java.util.concurrent.FutureTask.run(FutureTask.java:264) *09:45:25* at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) *09:45:25* at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) *09:45:25* at hudson.remoting.Engine$1$1.run(Engine.java:98) *09:45:25* at java.lang.Thread.run(Thread.java:844) Full log: [https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-macos-10.13-py3-build-test/2797//console]
Retrying does not resolve the problem, however, subsequent builds on the same node often do succeed (for the case above, three hours later another build succeeded.)
The error is highly reminiscent of https://issues.jenkins-ci.org/browse/JENKINS-19453 but that issue was fixed in the Jenkins 1.x series, and this is a much more modern version of Jenkins. Additionally, the error doesn't seem to be persistent (in that it's not necessary to restart the worker to resolve the problem.)
BTW, this is not just an OS X slave problem; we've had it happen to Linux workers too (although the missing class is different): https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-linux-trusty-py2.7.9-build/3214/console
I'm not really sure how to go about making a reproducing test case. Let me know if you have any ideas.
The other case stack trace looks like this:
23:28:29 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-172-31-82-71.ec2.internal/172.31.82.71:42464 23:28:29 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) 23:28:29 at hudson.remoting.UserResponse.retrieve(UserRequest.java:310) 23:28:29 at hudson.remoting.Channel.call(Channel.java:908) 23:28:29 at hudson.FilePath.act(FilePath.java:986) 23:28:29 at hudson.FilePath.act(FilePath.java:975) 23:28:29 at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137) 23:28:29 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:758) 23:28:29 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:749) 23:28:29 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1117) 23:28:29 at hudson.scm.SCM.checkout(SCM.java:495) 23:28:29 at hudson.model.AbstractProject.checkout(AbstractProject.java:1202) 23:28:29 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) 23:28:29 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 23:28:29 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) 23:28:29 at hudson.model.Run.execute(Run.java:1724) 23:28:29 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 23:28:29 at hudson.model.ResourceController.execute(ResourceController.java:97) 23:28:29 at hudson.model.Executor.run(Executor.java:429) 23:28:29 java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy10 23:28:29 at sun.reflect.GeneratedConstructorAccessor16.newInstance(Unknown Source) 23:28:29 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 23:28:29 at java.lang.reflect.Constructor.newInstance(Constructor.java:423) 23:28:29 at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) 23:28:29 at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:165) 23:28:29 at hudson.remoting.Channel.export(Channel.java:722) 23:28:29 at hudson.remoting.Channel.export(Channel.java:686) 23:28:29 at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:198) 23:28:29 at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) 23:28:29 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 23:28:29 at java.lang.reflect.Method.invoke(Method.java:498) 23:28:29 at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1218) 23:28:29 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136) 23:28:29 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 23:28:29 at hudson.remoting.UserRequest._serialize(UserRequest.java:250) 23:28:29 at hudson.remoting.UserRequest.serialize(UserRequest.java:259) 23:28:29 at hudson.remoting.UserRequest.perform(UserRequest.java:221) 23:28:29 at hudson.remoting.UserRequest.perform(UserRequest.java:53) 23:28:29 at hudson.remoting.Request$2.run(Request.java:358) 23:28:29 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 23:28:29 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 23:28:29 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 23:28:29 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 23:28:29 at hudson.remoting.Engine$1$1.run(Engine.java:94) 23:28:29 at java.lang.Thread.run(Thread.java:748) 23:28:29 Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-172-31-82-71.ec2.internal/172.31.82.71:42464 failed 23:28:29 at hudson.remoting.Channel.call(Channel.java:916) 23:28:29 at hudson.FilePath.act(FilePath.java:986) 23:28:29 Caused: java.io.IOException: remote file operation failed: /var/lib/jenkins/workspace/pytorch-builds/pytorch-linux-trusty-py2.7.9-build at hudson.remoting.Channel@54638415:JNLP4-connect connection from ip-172-31-82-71.ec2.internal/172.31.82.71:42464 23:28:29 at hudson.FilePath.act(FilePath.java:993) 23:28:29 at hudson.FilePath.act(FilePath.java:975) 23:28:29 at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137) 23:28:29 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:758) 23:28:29 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:749) 23:28:29 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1117) 23:28:29 at hudson.scm.SCM.checkout(SCM.java:495) 23:28:29 at hudson.model.AbstractProject.checkout(AbstractProject.java:1202) 23:28:29 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) 23:28:29 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) 23:28:29 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) 23:28:29 at hudson.model.Run.execute(Run.java:1724) 23:28:29 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 23:28:29 at hudson.model.ResourceController.execute(ResourceController.java:97) 23:28:29 at hudson.model.Executor.run(Executor.java:429)
- is related to
-
-
- Closed
-
-
-
- Closed
-
