Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29647

java.lang.NoClassDefFoundError thrown by jenkins slave while the class is in the class path.

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      0. Jenkins slave run for couple days.
      1. Git fetches failed one day suddenly.
      2. Complaint Putty class java.lang.NoClassDefFoundError. although it exists under ~/.jenkins/cache/jars in that slave.

      I assume the jenkins cache is messed up somehow or at least slave process suddenly has wrong memory of whats in the cache.
      so I did below test.
      1. disconnect the jenkins slave to kill slave java process.
      2. rm -rf ~/.jenkins/cache/jars.
      3. bring jenkins slave back online(launche method ssh)
      4. I did see the ~/.jenkins/cache/jars filled with jars. but it does not include
      PuTTYKey and it does not even include class JavaVMArguments.class.
      5. if manually copied jar files in another functional jenkins slave to the cache and re-launche agent, those jars/classes still cant be recognized by slave java process.

      Note: the master jvm uses jdk1.7.75, the slave uses jdk1.7.79.

      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - hudson.plugins.git.GitException: Failed to fetch from git@XXX:YYY.git
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:735)
      12:42:12 - at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:983)
      12:42:12 - at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1016)
      12:42:12 - at hudson.scm.SCM.checkout(SCM.java:488)
      12:42:12 - at hudson.model.AbstractProject.checkout(AbstractProject.java:1257)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      12:42:12 - at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      12:42:12 - at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      12:42:12 - at hudson.model.Run.execute(Run.java:1745)
      12:42:12 - at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      12:42:12 - at hudson.model.ResourceController.execute(ResourceController.java:89)
      12:42:12 - at hudson.model.Executor.run(Executor.java:240)
      12:42:12 - Caused by: hudson.plugins.git.GitException: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:172)
      12:42:12 - at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
      12:42:12 - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      12:42:12 - at java.lang.reflect.Method.invoke(Method.java:606)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      12:42:12 - at com.sun.proxy.$Proxy48.execute(Unknown Source)
      12:42:12 - at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:733)
      12:42:12 - ... 11 more
      12:42:12 - Caused by: java.io.IOException: Remote call on a004999.com failed
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:760)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      12:42:12 - ... 17 more
      12:42:12 - Caused by: java.lang.NoClassDefFoundError: org/kohsuke/putty/PuTTYKey
      12:42:12 - at com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey.getPrivateKeys(BasicSSHUserPrivateKey.java:132)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.createSshKeyFile(CliGitAPIImpl.java:1417)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1295)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:86)
      12:42:12 - at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:324)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      12:42:12 - at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:121)
      12:42:12 - at hudson.remoting.UserRequest.perform(UserRequest.java:49)
      12:42:12 - at hudson.remoting.Request$2.run(Request.java:324)
      12:42:12 - at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      12:42:12 - at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      12:42:12 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      12:42:12 - at java.lang.Thread.run(Thread.java:745)
      12:42:12 - at ......remote call to a004999.com(Native Method)
      12:42:12 - at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
      12:42:12 - at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
      12:42:12 - at hudson.remoting.Channel.call(Channel.java:752)
      12:42:12 - ... 18 more
      12:42:12 - ERROR: Error fetching remote repo 'origin'
      12:42:12 - Recording test results

        Attachments

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-21520 NoClassDefFoundError in git client 1.6.1

          • Major - Major loss of function.
          • Closed

            Activity

            Ascending order - Click to sort in descending order
            4 older comments
            Hide
            Permalink
            markewaite Mark Waite added a comment -

            I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins.

            I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.

            Show
            markewaite Mark Waite added a comment - I agree that the problem is a core problem, not a git client plugin or git plugin problem. I offered the git client plugin change and git plugin change as indicators that there may be a work-around by using a newer version of the git plugin and git client plugin, along with a newer version of Jenkins. I don't know the details of the Jenkins slave jar caching. I don't know anyone else who knows the details of the slave jar caching in that old Jenkins version, other than the original developers.
            Hide
            Permalink
            eshen ellen shen added a comment -

            I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection.
            so whatever shows up in the default jar cache may or may not apply to ssh agent connection.

            checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there.
            how does ssh-agent slave get the required classes from?

            Show
            eshen ellen shen added a comment - I checked the default jar cache code, seems it applies to jnlp type of connection, not ssh-agent type of connection. so whatever shows up in the default jar cache may or may not apply to ssh agent connection. checking heapdump from the node failed to fetch code, the n classes such as puttykey is really missing there. how does ssh-agent slave get the required classes from?
            Hide
            Permalink
            dcaroest David Caro added a comment -

            I'm having a similar issue with jenkins 1.656 and ssh agent 1.10:

            Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader

            I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest):

            Implementation-Version: 1.54.0

            And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version.
            I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas?

            Show
            dcaroest David Caro added a comment - I'm having a similar issue with jenkins 1.656 and ssh agent 1.10: Caused by: java.lang.ClassNotFoundException: org.bouncycastle.openssl.PEMReader I can see though that the bouncycastle jars downloaded on the master (/var/lib/data/jenkins/plugins/ssh-agent/WEB-INF/lib/bcprov-jdk15on-1.47.jar) are version 1.47, but the jars on the slave's .jenkins/cache/jars are 1.54 (latest): Implementation-Version: 1.54.0 And yes, if I check the classes that are included in both jars, the PEMReader is only available in the old version. I have no other plugins declaring any dependency on bouncycastle, and removing the cache and reconnecting the slave redownloads the newer version to it. Any ideas?
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            I see nothing in the stack trace pertaining to ssh-agent.

            Show
            jglick Jesse Glick added a comment - I see nothing in the stack trace pertaining to ssh-agent .
            Hide
            Permalink
            stephenconnolly Stephen Connolly added a comment -

            Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

            Show
            stephenconnolly Stephen Connolly added a comment - Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              eshen ellen shen
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated: