Details
-
Type:
Bug
-
Status: Open (View Workflow)
-
Priority:
Blocker
-
Resolution: Unresolved
-
Component/s: ssh-credentials-plugin
-
Labels:None
-
Similar Issues:
Description
0. Jenkins slave run for couple days.
1. Git fetches failed one day suddenly.
2. Complaint Putty class java.lang.NoClassDefFoundError. although it exists under ~/.jenkins/cache/jars in that slave.
I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
so I did below test.
1. disconnect the jenkins slave to kill slave java process.
2. rm -rf ~/.jenkins/cache/jars.
3. bring jenkins slave back online(launche method ssh)
4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
PuTTYKey and it does not even include class JavaVMArguments.class.
5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.
Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.
12:42:12 - ERROR: Error fetching remote repo 'origin'
12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
12:42:12 - at hudson.model.Run.execute(Run.java:1745)
12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
12:42:12 - at hudson.model.Executor.run(Executor.java:240)
12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
12:42:12 - ... 11 more
12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
12:42:12 - ... 17 more
12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
12:42:12 - at java.lang.Thread.run(Thread.java:745)
12:42:12 - at ......remote call to a004999.com(Native Method)
12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
12:42:12 - ... 18 more
12:42:12 - ERROR: Error fetching remote repo 'origin'
12:42:12 - Recording test results
Attachments
Issue Links
- is related to
-
-
- Closed
-
Activity
I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection.
so whatever shows up in the default jar cache may or may not apply to ssh agent connection.
checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there.
how does ssh-agent slave get the required classes from?
ellen shen
added a comment - I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection.
so whatever shows up in the default jar cache may or may not apply to ssh agent connection.
checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there.
how does ssh-agent slave get the required classes from? I'm having a similar issue with jenkins 1.656 and ssh agent 1.10:
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader
I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest):
Implementation-Version: 1.54.0
And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version.
I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas?
David Caro
added a comment - I'm having a similar issue with jenkins 1.656 and ssh agent 1.10:
Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader
I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest):
Implementation-Version: 1.54.0
And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version.
I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas? I see nothing in the stack trace pertaining to ssh-agent.
Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA
I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins.
I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.