Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29647

java.lang.NoClassDefFoundError thrown by jenkins slave while the class is in the class path.

XMLWordPrintable

      0. Jenkins slave run for couple days.
      1. Git fetches failed one day suddenly.
      2. Complaint Putty class java.lang.NoClassDefFoundError. although it exists under ~/.jenkins/cache/jars in that slave.

      I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
      so I did below test.
      1. disconnect the jenkins slave to kill slave java process.
      2. rm -rf ~/.jenkins/cache/jars.
      3. bring jenkins slave back online(launche method ssh)
      4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
      PuTTYKey and it does not even include class JavaVMArguments.class.
      5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.

      Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.

      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
      12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
      12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
      12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
      12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      12:42:12 - at hudson.model.Run.execute(Run.java:1745)
      12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
      12:42:12 - at hudson.model.Executor.run(Executor.java:240)
      12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
      12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
      12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
      12:42:12 - ... 11 more
      12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      12:42:12 - ... 17 more
      12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
      12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
      12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      12:42:12 - at java.lang.Thread.run(Thread.java:745)
      12:42:12 - at ......remote call to a004999.com(Native Method)
      12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
      12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
      12:42:12 - ... 18 more
      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - Recording test results

          [JENKINS-29647] java.lang.NoClassDefFoundError thrown by jenkins slave while the class is in the class path.

          ellen shen created issue -

          ellen shen added a comment -

          jenkins version: 1.538
          org.jenkins-ci.plugins:ssh-agent:1.5
          org.jenkins-ci.plugins:git:2.3.5
          org.jenkins-ci.plugins:git-client:1.16.1

          ellen shen added a comment - jenkins version: 1.538 org.jenkins-ci.plugins:ssh-agent:1.5 org.jenkins-ci.plugins:git:2.3.5 org.jenkins-ci.plugins:git-client:1.16.1
          ellen shen made changes -
          Priority Original: Minor [ 4 ] New: Blocker [ 1 ]
          ellen shen made changes -
          Description Original: 0. Jenkins slave run for couple days.
          1. Git fetches failed one day suddenly.
          2. Complaint Putty class java.lang.NoClassDefFoundError. although I found that under ~/.jenkins/cache/jars

          I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
          so I did below test.
          1. disconnect the jenkins slave to kill slave java process.
          2. rm -rf ~/.jenkins/cache/jars.
          3. bring jenkins slave back online(launche method ssh)
          4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
          PuTTYKey and it does not even include class JavaVMArguments.class.
          5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.

          Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.

          12:42:12 - ERROR: Error fetching remote repo 'origin'
          12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
          12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
          12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
          12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
          12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
          12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
          12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
          12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          12:42:12 - at hudson.model.Run.execute(Run.java:1745)
          12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
          12:42:12 - at hudson.model.Executor.run(Executor.java:240)
          12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
          12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
          12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
          12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
          12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
          12:42:12 - ... 11 more
          12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
          12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
          12:42:12 - ... 17 more
          12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
          12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
          12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
          12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          12:42:12 - at java.lang.Thread.run(Thread.java:745)
          12:42:12 - at ......remote call to a004999.com(Native Method)
          12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
          12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
          12:42:12 - ... 18 more
          12:42:12 - ERROR: Error fetching remote repo 'origin'
          12:42:12 - Recording test results           		New: 0. Jenkins slave run for couple days.
          1. Git fetches failed one day suddenly.
          2. Complaint Putty class java.lang.NoClassDefFoundError. although it exists under ~/.jenkins/cache/jars in that slave.

          I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
          so I did below test.
          1. disconnect the jenkins slave to kill slave java process.
          2. rm -rf ~/.jenkins/cache/jars.
          3. bring jenkins slave back online(launche method ssh)
          4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
          PuTTYKey and it does not even include class JavaVMArguments.class.
          5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.

          Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.

          12:42:12 - ERROR: Error fetching remote repo 'origin'
          12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
          12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
          12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
          12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
          12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
          12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
          12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
          12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          12:42:12 - at hudson.model.Run.execute(Run.java:1745)
          12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
          12:42:12 - at hudson.model.Executor.run(Executor.java:240)
          12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
          12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
          12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
          12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
          12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
          12:42:12 - ... 11 more
          12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
          12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
          12:42:12 - ... 17 more
          12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
          12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
          12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
          12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
          12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
          12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
          12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
          12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          12:42:12 - at java.lang.Thread.run(Thread.java:745)
          12:42:12 - at ......remote call to a004999.com(Native Method)
          12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
          12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
          12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
          12:42:12 - ... 18 more
          12:42:12 - ERROR: Error fetching remote repo 'origin'
          12:42:12 - Recording test results
          Mark Waite made changes -
          Link New: This issue is related to JENKINS-21520 [ JENKINS-21520 ]

          Mark Waite added a comment - - edited

          The git plugin 2.4.0 release and the git client plugin 1.18.0 release include a work around for JENKINS-21520 which has similar symptoms to the problem you're describing. The change is not a fix, but it was helpful in the case which ndeloof found that caused the symptom you're describing to appear consistently while he was developing some other changes.

          Unfortunately, your Jenkins version is too old to use that new version of the git plugin or the git client plugin. Since 1.538 is not a long term support version, and is more than 18 months old, you may want to evaluate updating to a newer long term support version.

          If that isn't workable, you could backport the changes made for git plugin 2.4.0 to a branch you would create based on git plugin 2.3.5. I don't plan to perform that backport because the current Jenkins base release for the git plugin (1.580) is already 10 months old and provides as much historical compatibility as I believe is reasonable.

          Mark Waite added a comment - - edited The git plugin 2.4.0 release and the git client plugin 1.18.0 release include a work around for JENKINS-21520 which has similar symptoms to the problem you're describing. The change is not a fix, but it was helpful in the case which ndeloof found that caused the symptom you're describing to appear consistently while he was developing some other changes. Unfortunately, your Jenkins version is too old to use that new version of the git plugin or the git client plugin. Since 1.538 is not a long term support version, and is more than 18 months old, you may want to evaluate updating to a newer long term support version. If that isn't workable, you could backport the changes made for git plugin 2.4.0 to a branch you would create based on git plugin 2.3.5. I don't plan to perform that backport because the current Jenkins base release for the git plugin (1.580) is already 10 months old and provides as much historical compatibility as I believe is reasonable.

          ellen shen added a comment -

          markewaite
          Thanks for reply.
          My issue is different from JENKINS-21520.
          The missing class is not gitclient class, it happened to
          java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
          and it happened suddenly after working for couple days.
          and the class claimed to be missing are actually in the jenkins slave's jar cache(./jenkins/cache/jars).

          If I disconnect slave, rm -rf jenkins slave jar cache and relaunch jenkins slave
          the class is really missing.
          manually add those jars in jenkins jar cache dose and re-launche jenkins slave does not help jenkins slave to find those jars.

          I believe its a more general problem which has nothing particular to do with git client plugins.

          where does slave's jenkins jar cache sync from? is it from somewhere in master?

          ellen shen added a comment - markewaite Thanks for reply. My issue is different from JENKINS-21520 . The missing class is not gitclient class, it happened to java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey and it happened suddenly after working for couple days. and the class claimed to be missing are actually in the jenkins slave's jar cache(./jenkins/cache/jars). If I disconnect slave, rm -rf jenkins slave jar cache and relaunch jenkins slave the class is really missing. manually add those jars in jenkins jar cache dose and re-launche jenkins slave does not help jenkins slave to find those jars. I believe its a more general problem which has nothing particular to do with git client plugins. where does slave's jenkins jar cache sync from? is it from somewhere in master?

          ellen shen added a comment -

          in my environment, ssh agent plugins and ssh credential plugins and ssh slaves plugins are all enabled.

          ellen shen added a comment - in my environment, ssh agent plugins and ssh credential plugins and ssh slaves plugins are all enabled.

          Mark Waite added a comment -

          I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins.

          I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.

          Mark Waite added a comment - I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins. I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.

          ellen shen added a comment -

          I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection.
          so whatever shows up in the default jar cache may or may not apply to ssh agent connection.

          checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there.
          how does ssh-agent slave get the required classes from?

          ellen shen added a comment - I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection. so whatever shows up in the default jar cache may or may not apply to ssh agent connection. checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there. how does ssh-agent slave get the required classes from?

            Unassigned Unassigned
            eshen ellen shen
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: