Details
-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Minor
-
Resolution: Not A Defect
-
Component/s: ssh-slaves-plugin
-
Labels:None
-
Environment:Jenkins ver. 2.150.2
ssh-slaves plugin version 1.27 to 1.29.4
-
Similar Issues:
Description
I noticed a severe performance degradation in the execution of the pipeline sh step when the agent is connected via the ssh-slaves plugin.
I performed extra testing on a standalone jenkins installed from docker image jenkins/jenkins:lts with a number of different versions of the ssh-slaves plugin.
Attached is the Jenkinsfile I used to test this.
With 10 iterations of the sh step, versions 1.26 takes 14-15 seconds while version 1.27 takes 24-27 seconds.
That is without taking into account the first build after the agent has started which always takes longer (22 seconds in 1.26 and 40 seconds in 1.27)
I have also tried "Launch agent via execution of a command on the master" which performs in a way comparable to the ssh-slaves plugin in version before 1.27.
The only workaround I found is to keep the ssh-slaves plugin at version 1.26.
Attachments
Attachments
Activity
Ivan Fernandez Calvo I tried setting the property, then disconnected and reconnected my agent. It makes no measurable difference.
I was unable to replicate the issue on a Docker environment using Jenkins ver. 2.164.3 and ssh-slaves 1.29.4, I will make another test with Jenkins ver. 2.150.2
Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on ssh-agent in /home/jenkins/workspace/test [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (One) [Pipeline] script [Pipeline] { [Pipeline] sh [2019-06-01T13:05:35.854Z] + /bin/bash -c time hostname [2019-06-01T13:05:35.855Z] [2019-06-01T13:05:35.855Z] real 0m0.001s [2019-06-01T13:05:35.855Z] user 0m0.000s [2019-06-01T13:05:35.855Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:35.967Z] cc9f8b5c29e1 [2019-06-01T13:05:35.967Z] [Pipeline] sh [2019-06-01T13:05:36.432Z] + /bin/bash -c time hostname [2019-06-01T13:05:36.434Z] [2019-06-01T13:05:36.434Z] real 0m0.004s [2019-06-01T13:05:36.434Z] user 0m0.000s [2019-06-01T13:05:36.434Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:36.536Z] cc9f8b5c29e1 [2019-06-01T13:05:36.537Z] [Pipeline] sh [2019-06-01T13:05:36.961Z] + /bin/bash -c time hostname [2019-06-01T13:05:36.961Z] [2019-06-01T13:05:36.961Z] real 0m0.001s [2019-06-01T13:05:36.961Z] user 0m0.000s [2019-06-01T13:05:36.961Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:37.069Z] cc9f8b5c29e1 [2019-06-01T13:05:37.069Z] [Pipeline] sh [2019-06-01T13:05:37.516Z] + /bin/bash -c time hostname [2019-06-01T13:05:37.516Z] [2019-06-01T13:05:37.516Z] real 0m0.002s [2019-06-01T13:05:37.516Z] user 0m0.000s [2019-06-01T13:05:37.516Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:37.554Z] cc9f8b5c29e1 [2019-06-01T13:05:37.555Z] [Pipeline] sh [2019-06-01T13:05:38.008Z] + /bin/bash -c time hostname [2019-06-01T13:05:38.008Z] [2019-06-01T13:05:38.008Z] real 0m0.001s [2019-06-01T13:05:38.009Z] user 0m0.000s [2019-06-01T13:05:38.009Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:38.148Z] cc9f8b5c29e1 [2019-06-01T13:05:38.148Z] [Pipeline] sh [2019-06-01T13:05:38.736Z] + /bin/bash -c time hostname [2019-06-01T13:05:38.737Z] [2019-06-01T13:05:38.737Z] real 0m0.001s [2019-06-01T13:05:38.737Z] user 0m0.000s [2019-06-01T13:05:38.737Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:38.756Z] cc9f8b5c29e1 [2019-06-01T13:05:38.756Z] [Pipeline] sh [2019-06-01T13:05:39.342Z] + /bin/bash -c time hostname [2019-06-01T13:05:39.342Z] [2019-06-01T13:05:39.342Z] real 0m0.001s [2019-06-01T13:05:39.342Z] user 0m0.000s [2019-06-01T13:05:39.342Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:39.447Z] cc9f8b5c29e1 [2019-06-01T13:05:39.448Z] [Pipeline] sh [2019-06-01T13:05:39.826Z] + /bin/bash -c time hostname [2019-06-01T13:05:39.826Z] [2019-06-01T13:05:39.826Z] real 0m0.001s [2019-06-01T13:05:39.826Z] user 0m0.000s [2019-06-01T13:05:39.826Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:39.846Z] cc9f8b5c29e1 [2019-06-01T13:05:39.847Z] [Pipeline] sh [2019-06-01T13:05:40.324Z] + /bin/bash -c time hostname [2019-06-01T13:05:40.324Z] [2019-06-01T13:05:40.324Z] real 0m0.001s [2019-06-01T13:05:40.324Z] user 0m0.000s [2019-06-01T13:05:40.324Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:40.354Z] cc9f8b5c29e1 [2019-06-01T13:05:40.354Z] [Pipeline] sh [2019-06-01T13:05:40.826Z] + /bin/bash -c time hostname [2019-06-01T13:05:40.826Z] [2019-06-01T13:05:40.826Z] real 0m0.006s [2019-06-01T13:05:40.826Z] user 0m0.000s [2019-06-01T13:05:40.826Z] sys 0m0.000s [Pipeline] echo [2019-06-01T13:05:40.913Z] cc9f8b5c29e1 [2019-06-01T13:05:40.914Z] [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
yep, something happens on 2.150.2, it does not have a consistent time of execution it differed about 5-10 secs, but it is not related with the ssh-slaves plugin and it seems resolved on 2.164.3
it has taken 9 sec
Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on ssh-agent in /home/jenkins/workspace/test [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (One) [Pipeline] script [Pipeline] { [Pipeline] sh [2019-06-01T14:24:31.793Z] + /bin/bash -c time hostname [2019-06-01T14:24:31.799Z] [2019-06-01T14:24:31.799Z] real 0m0.001s [2019-06-01T14:24:31.799Z] user 0m0.000s [2019-06-01T14:24:31.800Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:32.509Z] cc32194b98d6 [2019-06-01T14:24:32.509Z] [Pipeline] sh [2019-06-01T14:24:32.993Z] + /bin/bash -c time hostname [2019-06-01T14:24:32.994Z] [2019-06-01T14:24:32.994Z] real 0m0.001s [2019-06-01T14:24:32.994Z] user 0m0.000s [2019-06-01T14:24:32.994Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:33.102Z] cc32194b98d6 [2019-06-01T14:24:33.103Z] [Pipeline] sh [2019-06-01T14:24:33.965Z] + /bin/bash -c time hostname [2019-06-01T14:24:33.965Z] [2019-06-01T14:24:33.965Z] real 0m0.001s [2019-06-01T14:24:33.965Z] user 0m0.000s [2019-06-01T14:24:33.965Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:34.008Z] cc32194b98d6 [2019-06-01T14:24:34.009Z] [Pipeline] sh [2019-06-01T14:24:34.403Z] + /bin/bash -c time hostname [2019-06-01T14:24:34.404Z] [2019-06-01T14:24:34.404Z] real 0m0.001s [2019-06-01T14:24:34.404Z] user 0m0.000s [2019-06-01T14:24:34.404Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:34.500Z] cc32194b98d6 [2019-06-01T14:24:34.500Z] [Pipeline] sh [2019-06-01T14:24:35.185Z] + /bin/bash -c time hostname [2019-06-01T14:24:35.185Z] [2019-06-01T14:24:35.185Z] real 0m0.001s [2019-06-01T14:24:35.185Z] user 0m0.000s [2019-06-01T14:24:35.185Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:35.228Z] cc32194b98d6 [2019-06-01T14:24:35.228Z] [Pipeline] sh [2019-06-01T14:24:36.213Z] + /bin/bash -c time hostname [2019-06-01T14:24:36.213Z] [2019-06-01T14:24:36.213Z] real 0m0.002s [2019-06-01T14:24:36.213Z] user 0m0.000s [2019-06-01T14:24:36.213Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:36.600Z] cc32194b98d6 [2019-06-01T14:24:36.600Z] [Pipeline] sh [2019-06-01T14:24:37.111Z] + /bin/bash -c time hostname [2019-06-01T14:24:37.111Z] [2019-06-01T14:24:37.111Z] real 0m0.001s [2019-06-01T14:24:37.111Z] user 0m0.000s [2019-06-01T14:24:37.111Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:37.617Z] cc32194b98d6 [2019-06-01T14:24:37.617Z] [Pipeline] sh [2019-06-01T14:24:38.289Z] + /bin/bash -c time hostname [2019-06-01T14:24:38.289Z] [2019-06-01T14:24:38.289Z] real 0m0.001s [2019-06-01T14:24:38.289Z] user 0m0.000s [2019-06-01T14:24:38.289Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:38.396Z] cc32194b98d6 [2019-06-01T14:24:38.396Z] [Pipeline] sh [2019-06-01T14:24:38.889Z] + /bin/bash -c time hostname [2019-06-01T14:24:38.889Z] [2019-06-01T14:24:38.889Z] real 0m0.001s [2019-06-01T14:24:38.889Z] user 0m0.000s [2019-06-01T14:24:38.889Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:39.006Z] cc32194b98d6 [2019-06-01T14:24:39.006Z] [Pipeline] sh [2019-06-01T14:24:39.881Z] + /bin/bash -c time hostname [2019-06-01T14:24:39.881Z] [2019-06-01T14:24:39.881Z] real 0m0.002s [2019-06-01T14:24:39.881Z] user 0m0.000s [2019-06-01T14:24:39.881Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:24:40.280Z] cc32194b98d6 [2019-06-01T14:24:40.280Z] [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
it has taken 6 sec
Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on ssh-agent in /home/jenkins/workspace/test [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (One) [Pipeline] script [Pipeline] { [Pipeline] sh [2019-06-01T14:27:53.656Z] + /bin/bash -c time hostname [2019-06-01T14:27:53.657Z] [2019-06-01T14:27:53.657Z] real 0m0.002s [2019-06-01T14:27:53.657Z] user 0m0.000s [2019-06-01T14:27:53.657Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:53.860Z] cc32194b98d6 [2019-06-01T14:27:53.860Z] [Pipeline] sh [2019-06-01T14:27:54.280Z] + /bin/bash -c time hostname [2019-06-01T14:27:54.280Z] [2019-06-01T14:27:54.280Z] real 0m0.001s [2019-06-01T14:27:54.280Z] user 0m0.000s [2019-06-01T14:27:54.280Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:54.378Z] cc32194b98d6 [2019-06-01T14:27:54.378Z] [Pipeline] sh [2019-06-01T14:27:54.751Z] + /bin/bash -c time hostname [2019-06-01T14:27:54.751Z] [2019-06-01T14:27:54.751Z] real 0m0.001s [2019-06-01T14:27:54.751Z] user 0m0.000s [2019-06-01T14:27:54.751Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:55.047Z] cc32194b98d6 [2019-06-01T14:27:55.048Z] [Pipeline] sh [2019-06-01T14:27:55.527Z] + /bin/bash -c time hostname [2019-06-01T14:27:55.528Z] [2019-06-01T14:27:55.528Z] real 0m0.001s [2019-06-01T14:27:55.528Z] user 0m0.000s [2019-06-01T14:27:55.528Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:55.555Z] cc32194b98d6 [2019-06-01T14:27:55.555Z] [Pipeline] sh [2019-06-01T14:27:56.014Z] + /bin/bash -c time hostname [2019-06-01T14:27:56.014Z] [2019-06-01T14:27:56.014Z] real 0m0.002s [2019-06-01T14:27:56.014Z] user 0m0.000s [2019-06-01T14:27:56.014Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:56.053Z] cc32194b98d6 [2019-06-01T14:27:56.053Z] [Pipeline] sh [2019-06-01T14:27:56.468Z] + /bin/bash -c time hostname [2019-06-01T14:27:56.468Z] [2019-06-01T14:27:56.468Z] real 0m0.001s [2019-06-01T14:27:56.468Z] user 0m0.000s [2019-06-01T14:27:56.468Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:56.575Z] cc32194b98d6 [2019-06-01T14:27:56.575Z] [Pipeline] sh [2019-06-01T14:27:57.034Z] + /bin/bash -c time hostname [2019-06-01T14:27:57.034Z] [2019-06-01T14:27:57.034Z] real 0m0.005s [2019-06-01T14:27:57.034Z] user 0m0.000s [2019-06-01T14:27:57.034Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:57.079Z] cc32194b98d6 [2019-06-01T14:27:57.079Z] [Pipeline] sh [2019-06-01T14:27:57.544Z] + /bin/bash -c time hostname [2019-06-01T14:27:57.545Z] [2019-06-01T14:27:57.545Z] real 0m0.001s [2019-06-01T14:27:57.545Z] user 0m0.000s [2019-06-01T14:27:57.545Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:57.947Z] cc32194b98d6 [2019-06-01T14:27:57.948Z] [Pipeline] sh [2019-06-01T14:27:58.663Z] + /bin/bash -c time hostname [2019-06-01T14:27:58.664Z] [2019-06-01T14:27:58.664Z] real 0m0.001s [2019-06-01T14:27:58.664Z] user 0m0.000s [2019-06-01T14:27:58.664Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:58.785Z] cc32194b98d6 [2019-06-01T14:27:58.786Z] [Pipeline] sh [2019-06-01T14:27:59.261Z] + /bin/bash -c time hostname [2019-06-01T14:27:59.261Z] [2019-06-01T14:27:59.261Z] real 0m0.001s [2019-06-01T14:27:59.261Z] user 0m0.000s [2019-06-01T14:27:59.261Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:27:59.353Z] cc32194b98d6 [2019-06-01T14:27:59.353Z] [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
it has taken 3 sec
Started by user unknown or anonymous Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on ssh-agent in /home/jenkins/workspace/test [Pipeline] { [Pipeline] timestamps [Pipeline] { [Pipeline] stage [Pipeline] { (One) [Pipeline] script [Pipeline] { [Pipeline] sh [2019-06-01T14:31:30.418Z] + /bin/bash -c time hostname [2019-06-01T14:31:30.418Z] [2019-06-01T14:31:30.418Z] real 0m0.001s [2019-06-01T14:31:30.419Z] user 0m0.000s [2019-06-01T14:31:30.419Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:30.437Z] cc32194b98d6 [2019-06-01T14:31:30.437Z] [Pipeline] sh [2019-06-01T14:31:30.829Z] + /bin/bash -c time hostname [2019-06-01T14:31:30.829Z] [2019-06-01T14:31:30.829Z] real 0m0.001s [2019-06-01T14:31:30.829Z] user 0m0.000s [2019-06-01T14:31:30.829Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:30.850Z] cc32194b98d6 [2019-06-01T14:31:30.850Z] [Pipeline] sh [2019-06-01T14:31:31.152Z] + /bin/bash -c time hostname [2019-06-01T14:31:31.152Z] [2019-06-01T14:31:31.152Z] real 0m0.001s [2019-06-01T14:31:31.152Z] user 0m0.000s [2019-06-01T14:31:31.152Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:31.170Z] cc32194b98d6 [2019-06-01T14:31:31.170Z] [Pipeline] sh [2019-06-01T14:31:31.508Z] + /bin/bash -c time hostname [2019-06-01T14:31:31.508Z] [2019-06-01T14:31:31.508Z] real 0m0.001s [2019-06-01T14:31:31.508Z] user 0m0.000s [2019-06-01T14:31:31.508Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:31.527Z] cc32194b98d6 [2019-06-01T14:31:31.527Z] [Pipeline] sh [2019-06-01T14:31:31.823Z] + /bin/bash -c time hostname [2019-06-01T14:31:31.823Z] [2019-06-01T14:31:31.823Z] real 0m0.002s [2019-06-01T14:31:31.823Z] user 0m0.000s [2019-06-01T14:31:31.823Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:31.842Z] cc32194b98d6 [2019-06-01T14:31:31.843Z] [Pipeline] sh [2019-06-01T14:31:32.139Z] + /bin/bash -c time hostname [2019-06-01T14:31:32.139Z] [2019-06-01T14:31:32.139Z] real 0m0.001s [2019-06-01T14:31:32.139Z] user 0m0.000s [2019-06-01T14:31:32.139Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:32.157Z] cc32194b98d6 [2019-06-01T14:31:32.158Z] [Pipeline] sh [2019-06-01T14:31:32.625Z] + /bin/bash -c time hostname [2019-06-01T14:31:32.625Z] [2019-06-01T14:31:32.625Z] real 0m0.001s [2019-06-01T14:31:32.625Z] user 0m0.000s [2019-06-01T14:31:32.625Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:32.716Z] cc32194b98d6 [2019-06-01T14:31:32.716Z] [Pipeline] sh [2019-06-01T14:31:33.126Z] + /bin/bash -c time hostname [2019-06-01T14:31:33.126Z] [2019-06-01T14:31:33.126Z] real 0m0.002s [2019-06-01T14:31:33.126Z] user 0m0.000s [2019-06-01T14:31:33.126Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:33.148Z] cc32194b98d6 [2019-06-01T14:31:33.148Z] [Pipeline] sh [2019-06-01T14:31:33.522Z] + /bin/bash -c time hostname [2019-06-01T14:31:33.522Z] [2019-06-01T14:31:33.522Z] real 0m0.001s [2019-06-01T14:31:33.522Z] user 0m0.000s [2019-06-01T14:31:33.522Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:33.539Z] cc32194b98d6 [2019-06-01T14:31:33.539Z] [Pipeline] sh [2019-06-01T14:31:33.916Z] + /bin/bash -c time hostname [2019-06-01T14:31:33.917Z] [2019-06-01T14:31:33.917Z] real 0m0.001s [2019-06-01T14:31:33.917Z] user 0m0.000s [2019-06-01T14:31:33.917Z] sys 0m0.000s [Pipeline] echo [2019-06-01T14:31:33.939Z] cc32194b98d6 [2019-06-01T14:31:33.940Z] [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timestamps [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS
I've used the Docker compose environment defined on https://github.com/kuisathaverat/jenkins-issues/tree/master/JENKINS-57347
Ivan Fernandez Calvo I have been trying to use your docker compose setup to reproduce your success, but I am having a bit of trouble.
On the VM where I do this, port 22 is already used by a ssh server. So I added a couple of lines in the docker compose file to change the port to 8022 and when I start the jenkins and change the port (and the host name so it will reach my VM) in the ssh-agent node definition, it says it cannot connect with the key that is stored in the credentials.
you have to update the credential with the private key in the repo, the CasC configuration creates the key but the content is not correct because the Secret.kye is not the same.
Ok, I will try that now, where do I get the private key?
Probably related to track credentials, you can disable it-https://github.com/jenkinsci/ssh-slaves-plugin/blob/master/doc/TROUBLESHOOTING.md#threads-stuck-at-credentialsprovidertrackall-not related, I was thinking that the time was on the connection to the agent, I will make a test environment but the ssh-slave-plugin only manage the connection process and the way to open the channels, this should not be related to the time consumed by the steps, in any case, I will check it.