Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29647

java.lang.NoClassDefFoundError thrown by jenkins slave while the class is in the class path.

      0. Jenkins slave run for couple days.
      1. Git fetches failed one day suddenly.
      2. Complaint Putty class java.lang.NoClassDefFoundError. although it exists under ~/.jenkins/cache/jars in that slave.

      I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
      so I did below test.
      1. disconnect the jenkins slave to kill slave java process.
      2. rm -rf ~/.jenkins/cache/jars.
      3. bring jenkins slave back online(launche method ssh)
      4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
      PuTTYKey and it does not even include class JavaVMArguments.class.
      5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.

      Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.

      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
      12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
      12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
      12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
      12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      12:42:12 - at hudson.model.Run.execute(Run.java:1745)
      12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
      12:42:12 - at hudson.model.Executor.run(Executor.java:240)
      12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
      12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
      12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
      12:42:12 - ... 11 more
      12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      12:42:12 - ... 17 more
      12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
      12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
      12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      12:42:12 - at java.lang.Thread.run(Thread.java:745)
      12:42:12 - at ......remote call to a004999.com(Native Method)
      12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
      12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
      12:42:12 - ... 18 more
      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - Recording test results

          [JENKINS-29647] java.lang.NoClassDefFoundError thrown by jenkins slave while the class is in the class path.

          ellen shen added a comment -

          jenkins version: 1.538
          org.jenkins-ci.plugins:ssh-agent:1.5
          org.jenkins-ci.plugins:git:2.3.5
          org.jenkins-ci.plugins:git-client:1.16.1

          ellen shen added a comment - jenkins version: 1.538 org.jenkins-ci.plugins:ssh-agent:1.5 org.jenkins-ci.plugins:git:2.3.5 org.jenkins-ci.plugins:git-client:1.16.1

          Mark Waite added a comment - - edited

          The git plugin 2.4.0 release and the git client plugin 1.18.0 release include a work around for JENKINS-21520 which has similar symptoms to the problem you're describing. The change is not a fix, but it was helpful in the case which ndeloof found that caused the symptom you're describing to appear consistently while he was developing some other changes.

          Unfortunately, your Jenkins version is too old to use that new version of the git plugin or the git client plugin. Since 1.538 is not a long term support version, and is more than 18 months old, you may want to evaluate updating to a newer long term support version.

          If that isn't workable, you could backport the changes made for git plugin 2.4.0 to a branch you would create based on git plugin 2.3.5. I don't plan to perform that backport because the current Jenkins base release for the git plugin (1.580) is already 10 months old and provides as much historical compatibility as I believe is reasonable.

          Mark Waite added a comment - - edited The git plugin 2.4.0 release and the git client plugin 1.18.0 release include a work around for JENKINS-21520 which has similar symptoms to the problem you're describing. The change is not a fix, but it was helpful in the case which ndeloof found that caused the symptom you're describing to appear consistently while he was developing some other changes. Unfortunately, your Jenkins version is too old to use that new version of the git plugin or the git client plugin. Since 1.538 is not a long term support version, and is more than 18 months old, you may want to evaluate updating to a newer long term support version. If that isn't workable, you could backport the changes made for git plugin 2.4.0 to a branch you would create based on git plugin 2.3.5. I don't plan to perform that backport because the current Jenkins base release for the git plugin (1.580) is already 10 months old and provides as much historical compatibility as I believe is reasonable.

          ellen shen added a comment -

          markewaite
          Thanks for reply.
          My issue is different from JENKINS-21520.
          The missing class is not gitclient class, it happened to
          java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
          and it happened suddenly after working for couple days.
          and the class claimed to be missing are actually in the jenkins slave's jar cache(./jenkins/cache/jars).

          If I disconnect slave, rm -rf jenkins slave jar cache and relaunch jenkins slave
          the class is really missing.
          manually add those jars in jenkins jar cache dose and re-launche jenkins slave does not help jenkins slave to find those jars.

          I believe its a more general problem which has nothing particular to do with git client plugins.

          where does slave's jenkins jar cache sync from? is it from somewhere in master?

          ellen shen added a comment - markewaite Thanks for reply. My issue is different from JENKINS-21520 . The missing class is not gitclient class, it happened to java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey and it happened suddenly after working for couple days. and the class claimed to be missing are actually in the jenkins slave's jar cache(./jenkins/cache/jars). If I disconnect slave, rm -rf jenkins slave jar cache and relaunch jenkins slave the class is really missing. manually add those jars in jenkins jar cache dose and re-launche jenkins slave does not help jenkins slave to find those jars. I believe its a more general problem which has nothing particular to do with git client plugins. where does slave's jenkins jar cache sync from? is it from somewhere in master?

          ellen shen added a comment -

          in my environment, ssh agent plugins and ssh credential plugins and ssh slaves plugins are all enabled.

          ellen shen added a comment - in my environment, ssh agent plugins and ssh credential plugins and ssh slaves plugins are all enabled.

          Mark Waite added a comment -

          I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins.

          I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.

          Mark Waite added a comment - I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins. I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.

          ellen shen added a comment -

          I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection.
          so whatever shows up in the default jar cache may or may not apply to ssh agent connection.

          checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there.
          how does ssh-agent slave get the required classes from?

          ellen shen added a comment - I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection. so whatever shows up in the default jar cache may or may not apply to ssh agent connection. checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there. how does ssh-agent slave get the required classes from?

          David Caro added a comment -

          I'm having a similar issue with jenkins 1.656 and ssh agent 1.10:

          Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader

          I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest):

          Implementation-Version: 1.54.0

          And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version.
          I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas?

          David Caro added a comment - I'm having a similar issue with jenkins 1.656 and ssh agent 1.10: Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest): Implementation-Version: 1.54.0 And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version. I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas?

          Jesse Glick added a comment -

          I see nothing in the stack trace pertaining to ssh-agent.

          Jesse Glick added a comment - I see nothing in the stack trace pertaining to ssh-agent .

          Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

          Stephen Connolly added a comment - Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

            Unassigned Unassigned
            eshen ellen shen
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: