Critical Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels.
Since upgrading to EC2 plugin 1.49 (and to Jenkins 2.217 which contains remoting 4.0) some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true.
Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history.
Attachment: I created a snapshot about a node's script console page while - according to the Jenkins logs - it was used for building. I asked for the hostname and although the name of the node suggests it is a slave node, the hostname belongs to the master. And of course the workspace was created on master.
[JENKINS-61051] Jobs are started on master instead of EC2 slaves randomly
Gabor V
created issue -
Gabor V
made changes -
| Attachment | New: Screenshot 2020-02-11 at 13.59.23.png [ 50188 ] | |
| Description |
Original:
Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels. Since upgrading to EC2 plugin 1.49 some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true. Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history. |
New:
Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels. Since upgrading to EC2 plugin 1.49 some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true. Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history. Attachment: I created a snapshot about a node's script console page while - according to the Jenkins logs - it was used for building. I asked for the hostname and although the name of the node suggests it is a slave node, the hostname belongs to the master. And of course the workspace was created on master. |
Gabor V
made changes -
| Assignee | Original: FABRIZIO MANFREDI [ thoulen ] | New: Jeff Thompson [ jthompson ] |
Gabor V
made changes -
| Component/s | New: remoting [ 15489 ] | |
| Labels | Original: agents ec2 plugin slave | New: agents ec2 plugin remoting slave |
Gabor V
made changes -
| Description |
Original:
Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels. Since upgrading to EC2 plugin 1.49 some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true. Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history. Attachment: I created a snapshot about a node's script console page while - according to the Jenkins logs - it was used for building. I asked for the hostname and although the name of the node suggests it is a slave node, the hostname belongs to the master. And of course the workspace was created on master. |
New:
Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels. Since upgrading to EC2 plugin 1.49 (and to Jenkins 2.217 which contains remoting 4.0) some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true. Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history. Attachment: I created a snapshot about a node's script console page while - according to the Jenkins logs - it was used for building. I asked for the hostname and although the name of the node suggests it is a slave node, the hostname belongs to the master. And of course the workspace was created on master. |
Jeff Thompson
made changes -
| Assignee | Original: Jeff Thompson [ jthompson ] |
Gabor V
added a comment - "something about the ec2-plugin not launching the job in the right place" - I thought Jenkins launches the job, the ec2-plugin just creates the slaves
Gabor V
added a comment - "something about the ec2-plugin not launching the job in the right place" - I thought Jenkins launches the job, the ec2-plugin just creates the slaves Gabor V
made changes -
| Assignee | New: Francis Upton [ francisu ] |
Jeff Thompson
added a comment - I'm not familiar with the details of the ec2-plugin, but I know it does some complicated stuff, including with how it manages agents. When I've looked into the code there before, there was some complicated pieces. If you can reproduce the problems without the ec2-plugin then it probably is due to something in the Jenkins server. (Since I'm not familiar with any reports on that, it seems unlikely.) If it only occurs with the ec2-plugin, then it's probably something to do with the custom capabilities it provides.
Jeff Thompson
added a comment - I'm not familiar with the details of the ec2-plugin, but I know it does some complicated stuff, including with how it manages agents. When I've looked into the code there before, there was some complicated pieces. If you can reproduce the problems without the ec2-plugin then it probably is due to something in the Jenkins server. (Since I'm not familiar with any reports on that, it seems unlikely.) If it only occurs with the ec2-plugin, then it's probably something to do with the custom capabilities it provides. 
This probably doesn't have anything to do with Remoting. It's probably something about the ec2-plugin not launching the job in the right place or using the desired agent configuration. My guess is that it's going to require additional diagnostics in order to track this down. Anything you can do to collect better troubleshooting data or a reproducible scenario would likely be necessary to resolving this.