Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21518

Error while using SSH credentials to fetch git repository.

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      System is configured to use an SSH agent to get the build from the Git server, and build on the build agent. The system works intermittently, and when it doesn't, the error reported is:
      Started by an SCM change
      Started by an SCM change
      Started by an SCM change
      Building remotely on Build Agent 1 in workspace C:\Jenkins\workspace\Fusion 3.69
      FATAL: java.io.IOException: Remote call on Build Agent 1 failed
      hudson.remoting.RemotingSystemException: java.io.IOException: Remote call on Build Agent 1 failed
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:183)
      at com.sun.proxy.$Proxy75.addCredentials(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.addCredentials(RemoteGitImpl.java:196)
      at hudson.plugins.git.GitSCM.createClient(GitSCM.java:584)
      at hudson.plugins.git.GitSCM.createClient(GitSCM.java:561)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:866)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:557)
      at hudson.model.Run.execute(Run.java:1665)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: java.io.IOException: Remote call on Build Agent 1 failed
      at hudson.remoting.Channel.call(Channel.java:723)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:167)
      ... 13 more
      Caused by: java.lang.Error: Failed to deserialize the Callable object.
      at hudson.remoting.UserRequest.perform(UserRequest.java:104)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:58)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.ClassCastException: cannot assign instance of com.cloudbees.plugins.credentials.CredentialsScope$2 to field com.cloudbees.plugins.credentials.BaseCredentials.scope of type com.cloudbees.plugins.credentials.CredentialsScope in instance of com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
      at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(Unknown Source)
      at java.io.ObjectStreamClass.setObjFieldValues(Unknown Source)
      at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
      at java.io.ObjectInputStream.readSerialData(Unknown Source)
      at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
      at java.io.ObjectInputStream.readObject0(Unknown Source)
      at java.io.ObjectInputStream.readArray(Unknown Source)
      at java.io.ObjectInputStream.readObject0(Unknown Source)
      at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
      at java.io.ObjectInputStream.readSerialData(Unknown Source)
      at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
      at java.io.ObjectInputStream.readObject0(Unknown Source)
      at java.io.ObjectInputStream.readObject(Unknown Source)
      at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
      at hudson.remoting.UserRequest.perform(UserRequest.java:98)
      ... 9 more

      This seems similar to a few other SSH/Git related issues that are there, so any patches and/or pointers to fix this would be appreciated.

        Attachments

          Activity

          Hide
          shuwada Hiroshi Wada added a comment - - edited

          I've got the exactly same issue with the combination of Linux master (Jenkins 1.550 on Ubuntu 12.04) and Windows slave (slave.jar is up-to-date.) We manage private keys with Jenkins credentials plugin and directly paste keys on Jenkins console.

          Strange thing is remote build succeeds after rebooting a master or a slave; however, after while, all remote builds start failing. Not exactly sure what triggers the failure but it seems to occur after running a project which uses a different key, or after one of Jenkins projects fails due to wrong key settings.

          Show
          shuwada Hiroshi Wada added a comment - - edited I've got the exactly same issue with the combination of Linux master (Jenkins 1.550 on Ubuntu 12.04) and Windows slave (slave.jar is up-to-date.) We manage private keys with Jenkins credentials plugin and directly paste keys on Jenkins console. Strange thing is remote build succeeds after rebooting a master or a slave; however, after while, all remote builds start failing. Not exactly sure what triggers the failure but it seems to occur after running a project which uses a different key, or after one of Jenkins projects fails due to wrong key settings.
          Hide
          wvl0 William Leese added a comment -

          Same issue, Linux master & linux slaves. Jenkins 1.540, but with the credentials and ssh-credentials plugins up to date.

          Show
          wvl0 William Leese added a comment - Same issue, Linux master & linux slaves. Jenkins 1.540, but with the credentials and ssh-credentials plugins up to date.
          Hide
          markewaite Mark Waite added a comment -

          Have you checked with the most recent versions of the git-client-plugin and the git-plugin?

          I've not seen any issues with remoting failures using my collection of Debian Linux variants (7.1 x86, Testing x64, 7.1 x64) and Windows (Win 7 x86, Windows Home Server 2011 x86, Windows 8.1 x64). They are all running Java 1.7.0_51 as provided by Oracle, though I don't expect any issues related to the provider of the JDK either.

          Show
          markewaite Mark Waite added a comment - Have you checked with the most recent versions of the git-client-plugin and the git-plugin? I've not seen any issues with remoting failures using my collection of Debian Linux variants (7.1 x86, Testing x64, 7.1 x64) and Windows (Win 7 x86, Windows Home Server 2011 x86, Windows 8.1 x64). They are all running Java 1.7.0_51 as provided by Oracle, though I don't expect any issues related to the provider of the JDK either.
          Hide
          djs Dan Savilonis added a comment -

          I'm seeing the same behavior on 1.555 with up to date git plugins as of Mar 17. This was working fine and I kept hitting 'rebuild last', and suddenly I got this error. A regular build does not work either.

          I was editing the job, so I expect that somehow introduced this. I've seen job configuration get corrupted in the past after repeated saves.

          Show
          djs Dan Savilonis added a comment - I'm seeing the same behavior on 1.555 with up to date git plugins as of Mar 17. This was working fine and I kept hitting 'rebuild last', and suddenly I got this error. A regular build does not work either. I was editing the job, so I expect that somehow introduced this. I've seen job configuration get corrupted in the past after repeated saves.
          Hide
          metastable Meta Stable added a comment -

          I am currently running Jenkins 1.537, with the following plugins:
          Credentials plugin: 1.9.4
          SSH Credentials Plugin: 1.6
          SSH Slaves Plugin: 1.5
          Strictly speaking, I'm not sure any are required other than "SSH Credentials Plugin".

          I see that new version(s) are available, both for Jenkins as well as some of these plugins, so I will update and check. But I am still facing this problem.

          Show
          metastable Meta Stable added a comment - I am currently running Jenkins 1.537, with the following plugins: Credentials plugin: 1.9.4 SSH Credentials Plugin: 1.6 SSH Slaves Plugin: 1.5 Strictly speaking, I'm not sure any are required other than "SSH Credentials Plugin". I see that new version(s) are available, both for Jenkins as well as some of these plugins, so I will update and check. But I am still facing this problem.
          Hide
          markewaite Mark Waite added a comment -

          The Credentials plugin is required with the git-plugin and the git-client-plugin.

          Show
          markewaite Mark Waite added a comment - The Credentials plugin is required with the git-plugin and the git-client-plugin.
          Hide
          iflowfor8hours matt urbanski added a comment -

          I have seen this when a slave is using two sets of ssh credentials at the same time. In my case it was the ssh-agent plugin and the git plugin conflicting. As soon as I got rid of one of them, everything worked again.
          YMMV.

          Show
          iflowfor8hours matt urbanski added a comment - I have seen this when a slave is using two sets of ssh credentials at the same time. In my case it was the ssh-agent plugin and the git plugin conflicting. As soon as I got rid of one of them, everything worked again. YMMV.
          Hide
          blewa Aaron Blew added a comment -

          Same issue with swarm slaves, no SSH credentials handled via Jenkins. I can restart the master and it will build successfully. We've also had success building before the restart by removing the wildcards from the branches we want the job to build.

          This is kind of a painful one. Is there any other info we could provide that would help find a solution?

          Thanks!

          Show
          blewa Aaron Blew added a comment - Same issue with swarm slaves, no SSH credentials handled via Jenkins. I can restart the master and it will build successfully. We've also had success building before the restart by removing the wildcards from the branches we want the job to build. This is kind of a painful one. Is there any other info we could provide that would help find a solution? Thanks!
          Hide
          markewaite Mark Waite added a comment -

          Aaron Blew have you tried the suggestions listed earlier (confirm you're only using one set of ssh credentials, confirm the slave jar matches your Jenkins version, etc.)?

          Swarm slaves may also have an additional complexity because I believe they embed a copy of slave.jar in the swarm jar file. You might do a test drive with one of your swarm slaves and switch it temporarily to use Java web start or ssh to launch the slave. If that shows a different behavior, then your case may indicate that the swarm jar needs an update.

          Show
          markewaite Mark Waite added a comment - Aaron Blew have you tried the suggestions listed earlier (confirm you're only using one set of ssh credentials, confirm the slave jar matches your Jenkins version, etc.)? Swarm slaves may also have an additional complexity because I believe they embed a copy of slave.jar in the swarm jar file. You might do a test drive with one of your swarm slaves and switch it temporarily to use Java web start or ssh to launch the slave. If that shows a different behavior, then your case may indicate that the swarm jar needs an update.
          Hide
          blewa Aaron Blew added a comment -

          I just upgraded our swarm jar and haven't seen an issue in a few builds, but I'll keep an eye on it. We went from version 1.09 to version 1.20 (didn't realize it'd been updated so much between now and then).

          Thanks for the tip!

          Show
          blewa Aaron Blew added a comment - I just upgraded our swarm jar and haven't seen an issue in a few builds, but I'll keep an eye on it. We went from version 1.09 to version 1.20 (didn't realize it'd been updated so much between now and then). Thanks for the tip!
          Hide
          markewaite Mark Waite added a comment -

          Since I can't duplicate it and the comments on the bug have been quiet for over a month, I'm closing this bug as "Cannot Reproduce".

          Show
          markewaite Mark Waite added a comment - Since I can't duplicate it and the comments on the bug have been quiet for over a month, I'm closing this bug as "Cannot Reproduce".
          Hide
          zhangguoqing zhang guoqing added a comment -

          I maybe find the reason for this error. In my test bed, I have encounter the same problem that error likes the follows, when I build my jenkins project.

          FATAL: java.io.IOException: Remote call on <a-slave-host> failed
          hudson.remoting.RemotingSystemException: java.io.IOException: Remote call on <a-slave-host> failed
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:195)
          at com.sun.proxy.$Proxy75.addCredentials(Unknown Source)
          at org.jenkinsci.plugins.gitclient.RemoteGitImpl.addCredentials(RemoteGitImpl.java:197)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:662)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:638)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:978)
          at hudson.scm.SCM.checkout(SCM.java:484)
          at hudson.model.AbstractProject.checkout(AbstractProject.java:1265)
          at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
          at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
          at hudson.model.Run.execute(Run.java:1718)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:89)
          at hudson.model.Executor.run(Executor.java:240)
          Caused by: java.io.IOException: Remote call on rpmbuild-172.16.102.46 failed
          at hudson.remoting.Channel.call(Channel.java:760)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:179)
          ... 14 more

          At first, my jenkins project had fill the Git Repository URL with a type of Credentials. And I have put the <a-slave-host>'s id_rsa.pub which location '/root/.ssh/' to my gitlab server with 'Add SSH Key' operation.

          So you may known that the real reason. After above build errors, I set the Git Credentials to none by the jenkins server WEBUI for my project configure. Then I build the project successful. Without comment, we can get a conclusion that the multiple credential types for gitlab server is the real reason. So you known how to deal with the problem too.

          Thanks all.

          Show
          zhangguoqing zhang guoqing added a comment - I maybe find the reason for this error. In my test bed, I have encounter the same problem that error likes the follows, when I build my jenkins project. FATAL: java.io.IOException: Remote call on <a-slave-host> failed hudson.remoting.RemotingSystemException: java.io.IOException: Remote call on <a-slave-host> failed at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:195) at com.sun.proxy.$Proxy75.addCredentials(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.addCredentials(RemoteGitImpl.java:197) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:662) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:638) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:978) at hudson.scm.SCM.checkout(SCM.java:484) at hudson.model.AbstractProject.checkout(AbstractProject.java:1265) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528) at hudson.model.Run.execute(Run.java:1718) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:89) at hudson.model.Executor.run(Executor.java:240) Caused by: java.io.IOException: Remote call on rpmbuild-172.16.102.46 failed at hudson.remoting.Channel.call(Channel.java:760) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:179) ... 14 more At first, my jenkins project had fill the Git Repository URL with a type of Credentials. And I have put the <a-slave-host>'s id_rsa.pub which location '/root/.ssh/' to my gitlab server with 'Add SSH Key' operation. So you may known that the real reason. After above build errors, I set the Git Credentials to none by the jenkins server WEBUI for my project configure. Then I build the project successful. Without comment, we can get a conclusion that the multiple credential types for gitlab server is the real reason. So you known how to deal with the problem too. Thanks all.

            People

            Assignee:
            ndeloof Nicolas De Loof
            Reporter:
            metastable Meta Stable
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: