[JENKINS-64910] Hang when using ssh after upgrade to ssh-agent 1.21

Type: Bug
Resolution: Won't Fix
Priority: Blocker
Component/s: ssh-agent-plugin
Labels:
None
Environment:
Jenkins version 2.280
ssh-agent 1.21
ssh-credentials 1.18.1

Similar Issues:
Powered by SuggestiMate

Show

After upgrading to ssh-agent 1.21 we had a job hang at a git command.

We terminated it after more than 30 minutes. Tried again and again with same result.

After reverting to 1.20 it worked again.

Versions:

Jenkins 2.280
ssh-agent 1.21
ssh-credentials 1.18.1

Log:

...
[Pipeline] sshagent
15:27:55  [ssh-agent] Using credentials jenkins (jenkins ssh key)
15:27:55  [ssh-agent] Looking for ssh-agent implementation...
15:27:55  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
15:27:55  $ docker exec bb2595d46378e3aa4279cba0a20d8e3026f79d84bc9d2f2b5b856bbfba3ab1f5 ssh-agent
15:27:55  [ssh-agent]   Java/JNR ssh-agent
15:27:55  [ssh-agent] Registered BouncyCastle on the remote agent
15:27:55  [ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
15:27:55  + git ls-remote --heads origin

Probably related to JENKINS-50181.

relates to

JENKINS-43050 SSH Agent plugin doesn't work well with docker pipelines

Open

Robin Karlsson created issue - 2021-02-22 07:44

Vittorio added a comment - 2021-02-23 16:40

I have the exact same issue under Jenkins 2.263.4.

I use the SSH key for the agent as follows:

node { 
 sshagent(['deploy-dev']) {
  ... do something ...
 }
}

It works fine with SSH-Agent 1.20, but the build hangs till the pipeline's timeout with SSH-Agent 1.21.
I wonder if I should use the key like this instead:

node { 
 sshagent(credentials: ['deploy-dev']) {
  ... do something ...
 }
}

as documented here: https://www.jenkins.io/doc/pipeline/steps/ssh-agent/

I did not find anything related to this change between SSH-Agent 1.20 and 1.21 though.

Vittorio added a comment - 2021-02-23 16:40 I have the exact same issue under Jenkins 2.263.4. I use the SSH key for the agent as follows: node { sshagent([ 'deploy-dev' ]) { ... do something ... } } It works fine with SSH-Agent 1.20, but the build hangs till the pipeline's timeout with SSH-Agent 1.21. I wonder if I should use the key like this instead: node { sshagent(credentials: [ 'deploy-dev' ]) { ... do something ... } } as documented here: https://www.jenkins.io/doc/pipeline/steps/ssh-agent/ I did not find anything related to this change between SSH-Agent 1.20 and 1.21 though.

Mathieu added a comment - 2021-02-24 10:02

I have the same issue with Jenkins 2.281.

Looking at the agent.log file in jenkins home, I see the following error:

channel startedchannel started

Exception in thread "SSH Agent socket acceptor /var/jenkins_home/workspace/st_mbria_jenkins_ssh_agent_issue@tmp/ssh3694863879760087453" Exception in thread "SSH Agent socket acceptor /var/jenkins_home/workspace/st_mbria_jenkins_ssh_agent_issue@tmp/ssh3694863879760087453" java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of hudson/remoting/RemoteClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of hudson/remoting/RemoteClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signaturejava.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of hudson/remoting/RemoteClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of hudson/remoting/RemoteClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature

        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418)    at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418)

        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)

        at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:58)  at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:58)

        at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:43)  at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:43)

        at org.apache.sshd.agent.common.AbstractAgentClient.<init>(AbstractAgentClient.java:41) at org.apache.sshd.agent.common.AbstractAgentClient.<init>(AbstractAgentClient.java:41)

        at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentSessionSocketHandler.<init>(AgentServer.java:210)     at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentSessionSocketHandler.<init>(AgentServer.java:210)

        at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentServerSocketHandler.process(AgentServer.java:193)     at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentServerSocketHandler.process(AgentServer.java:193)

        at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$AgentSocketAcceptor.run(AgentServer.java:123) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$AgentSocketAcceptor.run(AgentServer.java:123)

        at java.lang.Thread.run(Thread.java:748)        at java.lang.Thread.run(Thread.java:748)

channel stoppedchannel stopped

I went back to version 1.20 and everything works fine so far.

Mathieu added a comment - 2021-02-24 10:02 I have the same issue with Jenkins 2.281. Looking at the agent.log file in jenkins home, I see the following error: channel startedchannel started Exception in thread "SSH Agent socket acceptor / var /jenkins_home/workspace/st_mbria_jenkins_ssh_agent_issue@tmp/ssh3694863879760087453" Exception in thread "SSH Agent socket acceptor / var /jenkins_home/workspace/st_mbria_jenkins_ssh_agent_issue@tmp/ssh3694863879760087453" java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of hudson/remoting/RemoteClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of hudson/remoting/RemoteClassLoader) for the method 's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signaturejava.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of hudson/remoting/RemoteClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of hudson/remoting/RemoteClassLoader) for the method' s defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:58) at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:58) at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:43) at org.apache.sshd.common.util.logging.AbstractLoggingBean.<init>(AbstractLoggingBean.java:43) at org.apache.sshd.agent.common.AbstractAgentClient.<init>(AbstractAgentClient.java:41) at org.apache.sshd.agent.common.AbstractAgentClient.<init>(AbstractAgentClient.java:41) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentSessionSocketHandler.<init>(AgentServer.java:210) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentSessionSocketHandler.<init>(AgentServer.java:210) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentServerSocketHandler.process(AgentServer.java:193) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$SshAgentServerSocketHandler.process(AgentServer.java:193) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$AgentSocketAcceptor.run(AgentServer.java:123) at com.cloudbees.jenkins.plugins.sshagent.jna.AgentServer$AgentSocketAcceptor.run(AgentServer.java:123) at java.lang. Thread .run( Thread .java:748) at java.lang. Thread .run( Thread .java:748) channel stoppedchannel stopped I went back to version 1.20 and everything works fine so far.

Vittorio added a comment - 2021-03-04 11:27 - edited

In reference to my previous comment, I have tried to use the SSH key as follows:

node { 
 sshagent(credentials: ['deploy-dev']) {
  ... do something ...
 }
}

It works fine on SSH Agent 1.20, but it still fails in the same way with SSH Agent 1.21: so I had to revert again to 1.20.

Vittorio added a comment - 2021-03-04 11:27 - edited In reference to my previous comment, I have tried to use the SSH key as follows: node { sshagent(credentials: [ 'deploy-dev' ]) { ... do something ... } } It works fine on SSH Agent 1.20, but it still fails in the same way with SSH Agent 1.21: so I had to revert again to 1.20.

Jesse Glick added a comment - 2021-03-10 16:42

From your log, you are using a deprecated in-Java implementation. Make sure the ssh-agent command is installed on your build node. See https://github.com/jenkinsci/ssh-agent-plugin/pull/48

Jesse Glick added a comment - 2021-03-10 16:42 From your log, you are using a deprecated in-Java implementation. Make sure the ssh-agent command is installed on your build node. See https://github.com/jenkinsci/ssh-agent-plugin/pull/48

Robin Karlsson added a comment - 2021-03-11 07:14

The build is using a docker agent with the image python:3.8

agent {
    docker {
        image 'python:3.8'
        args '-u root:root'
    }
}

As far as I can tell, that image has ssh-agent installed:

$ docker run --rm -it python:3.8 ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-8QKJYPNyxAfB/agent.1; export SSH_AUTH_SOCK;
SSH_AGENT_PID=8; export SSH_AGENT_PID;
echo Agent pid 8;

All hosts that run the Jenkins agents also have ssh-agent installed, if that matters.

Is there any way to get more info about why the Java-implementation was used and not the exec-based ssh-agent?

Robin Karlsson added a comment - 2021-03-11 07:14 The build is using a docker agent with the image python:3.8 agent { docker { image 'python:3.8' args '-u root:root' } } As far as I can tell, that image has ssh-agent installed: $ docker run --rm -it python:3.8 ssh-agent SSH_AUTH_SOCK=/tmp/ssh-8QKJYPNyxAfB/agent.1; export SSH_AUTH_SOCK; SSH_AGENT_PID=8; export SSH_AGENT_PID; echo Agent pid 8; All hosts that run the Jenkins agents also have ssh-agent installed, if that matters. Is there any way to get more info about why the Java-implementation was used and not the exec-based ssh-agent ?

Mathieu added a comment - 2021-03-11 07:46

I confirm the ssh-agent is also installed in the docker image used to build on my side.

This agent also works correctly: if I copy the SSH key in the docker instance, launch the agent and then try to do a git clone or connect to a SSH remote, everything works fine.

Mathieu added a comment - 2021-03-11 07:46 I confirm the ssh-agent is also installed in the docker image used to build on my side. This agent also works correctly: if I copy the SSH key in the docker instance, launch the agent and then try to do a git clone or connect to a SSH remote, everything works fine.

Jesse Glick added a comment - 2021-03-11 19:28

mbriand snago see JENKINS-43050; you cannot mix this plugin with docker-workflow. Frankly I recommend using neither.

Jesse Glick added a comment - 2021-03-11 19:28 mbriand snago see JENKINS-43050 ; you cannot mix this plugin with docker-workflow . Frankly I recommend using neither.

Jesse Glick made changes - 2021-03-11 19:28

Link

New: This issue relates to JENKINS-43050 [ JENKINS-43050 ]

Vittorio added a comment - 2021-03-19 10:22

jglick Thank you very much for your comment. I have solved the issue by installing the `ssh-agent` command in the container running the pipeline's stage, which uses the SSH Agent plugin.

So my pipeline is now working with the version 1.21.

Vittorio added a comment - 2021-03-19 10:22 jglick Thank you very much for your comment. I have solved the issue by installing the `ssh-agent` command in the container running the pipeline's stage, which uses the SSH Agent plugin. So my pipeline is now working with the version 1.21.

Assignee:: Unassigned

Reporter:: Robin Karlsson

Votes:: 2 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2021-02-22 07:44

Updated:: 2021-04-03 08:01

Resolved:: 2021-03-24 15:06

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Vittorio added a comment - 2021-02-23 16:40

Expand comment: Vittorio added a comment - 2021-02-23 16:40

Collapse comment: Mathieu added a comment - 2021-02-24 10:02

Expand comment: Mathieu added a comment - 2021-02-24 10:02

Collapse comment: Vittorio added a comment - 2021-03-04 11:27, Edited by Vittorio - 2021-03-04 15:16

Expand comment: Vittorio added a comment - 2021-03-04 11:27, Edited by Vittorio - 2021-03-04 15:16

Collapse comment: Jesse Glick added a comment - 2021-03-10 16:42

Expand comment: Jesse Glick added a comment - 2021-03-10 16:42

Collapse comment: Robin Karlsson added a comment - 2021-03-11 07:14

Expand comment: Robin Karlsson added a comment - 2021-03-11 07:14

Collapse comment: Mathieu added a comment - 2021-03-11 07:46

Expand comment: Mathieu added a comment - 2021-03-11 07:46

Collapse comment: Jesse Glick added a comment - 2021-03-11 19:28

Expand comment: Jesse Glick added a comment - 2021-03-11 19:28

Collapse comment: Vittorio added a comment - 2021-03-19 10:22

Expand comment: Vittorio added a comment - 2021-03-19 10:22

People

Dates