Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61051

Jobs are started on master instead of EC2 slaves randomly

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Jenkins master runs on an AWS Linux 2. Jenkins uses the EC2 plugin to create slaves whenever needed and many jobs are assigned to slaves using the labels.

      Since upgrading to EC2 plugin 1.49 (and to Jenkins 2.217 which contains remoting 4.0) some jobs - randomly, it seems - are started on the master node instead of using the started slaves. The aws slave is started, but the workspace is created on master (in the user's home which should have been used on the slave). The job's console log says it is running on the slave but it is not true.

      Maybe this is not related to EC2 plugin as I don't see any change related to this problem in the 1.49 version's release history.

      Attachment: I created a snapshot about a node's script console page while - according to the Jenkins logs - it was used for building. I asked for the hostname and although the name of the node suggests it is a slave node, the hostname belongs to the master. And of course the workspace was created on master.

        Attachments

          Activity

          Hide
          laszlog Laszlo Gaal added a comment -

          Just ran into this again. Jeff Thompson: yeah, it looks like either a timing problem or a race.

          As a workaround I installed roadblocks on the master that should fail such an errant job very early in the startup/config phase, before it has a chance to consume all memory and trigger an OOM-kill. We'll see if it's enough; I'd really hate to downgrade the plugin again.

          Show
          laszlog Laszlo Gaal added a comment - Just ran into this again. Jeff Thompson : yeah, it looks like either a timing problem or a race. As a workaround I installed roadblocks on the master that should fail such an errant job very early in the startup/config phase, before it has a chance to consume all memory and trigger an OOM-kill. We'll see if it's enough; I'd really hate to downgrade the plugin again.
          Hide
          gaborv Gabor V added a comment -

           Any idea who can work on this bug from the ec2 plugin team? To whom should we assign it?

          Show
          gaborv Gabor V added a comment -  Any idea who can work on this bug from the ec2 plugin team? To whom should we assign it?
          Hide
          raihaan Raihaan Shouhell added a comment -

          EC2 just launches and manages agents it doesn't actually do anything with regards to assigning agents.
          That null does look suspicious.

          Does your master use the same pem as your agents? I'm assuming that your agents are linux and using ssh as well.

          Show
          raihaan Raihaan Shouhell added a comment - EC2 just launches and manages agents it doesn't actually do anything with regards to assigning agents. That null does look suspicious. Does your master use the same pem as your agents? I'm assuming that your agents are linux and using ssh as well.
          Hide
          laszlog Laszlo Gaal added a comment -

          Raihaan Shouhell, yes, they do use the same keys, and I've realized that assigning different keys to them would be a useful workaround.

          However, I've never had this problem before upgrading to 1.49.1, so having the same keys does not caue the problem, although it makes the failing case that much more severe.

          Show
          laszlog Laszlo Gaal added a comment - Raihaan Shouhell , yes, they do use the same keys, and I've realized that assigning different keys to them would be a useful workaround. However, I've never had this problem before upgrading to 1.49.1, so having the same keys does not caue the problem, although it makes the failing case that much more severe.
          Hide
          laszlog Laszlo Gaal added a comment - - edited

          Just saw https://github.com/jenkinsci/ec2-plugin/pull/447, which seems likely to fix this issue; one of the comments actually refers to the

          Connecting to null on port 22 

          pattern I described in an earlier comment.

          Show
          laszlog Laszlo Gaal added a comment - - edited Just saw https://github.com/jenkinsci/ec2-plugin/pull/447 , which seems likely to fix this issue; one of the comments actually refers to the Connecting to null on port 22 pattern I described in an earlier comment.

            People

            Assignee:
            francisu Francis Upton
            Reporter:
            gaborv Gabor V
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: