Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55078

No longer able to start ECS slaves

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      v1.19

      Description

      We use the ECS plugin to start dynamic slaves when starting jobs. This worked fine, but suddenly stopped working.
      On the UI, all we get is: ‘Jenkins’ doesn’t have label <label>
      The different labels are still in the Jenkins settings however.
      If I look at the log it shows me the following, but it doesn't give any reason or explanation, that would help me understand, what could be causing this.
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure to run task with definition arn:aws:ecs:eu-central-1:x:task-definition/ECS-mgmt:7281 on ECS cluster arn:aws:ecs:eu-central-1:x:cluster/default
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/7b1c9cc6-7eb8-43fc-81cc-5a9d59d09d84
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/8f76d62a-398f-47b0-a883-50f329b41e8c
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/f3e2fddb-17ff-4fe4-9446-eaba4b05b298
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv] hudson.AbortException: Failed to run agent container ECS-wlxkv at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227) at com.cloudbees.jenkins.plugins.amazonecs.ECSLaunche
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch
      WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv]
      hudson.AbortException: Failed to run agent container ECS-wlxkv
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate INFO: [ECS-wlxkv]: Terminating ECS Task: null

        Attachments

          Issue Links

            Activity

            Hide
            autarchprinceps autarch princeps added a comment -

            I was able to fix the issue for me, by downgrading the plugin, so that seems to indicate a bug of some kind

            Show
            autarchprinceps autarch princeps added a comment - I was able to fix the issue for me, by downgrading the plugin, so that seems to indicate a bug of some kind
            Hide
            autarchprinceps autarch princeps added a comment -

            I spoke with the AWS Support and they said the problem was the addition of the attribute restraint ecs.capability.execution-role-awslogs. Previously the task definition only had com.amazonaws.ecs.capability.logging-driver.awslogs, which was sufficient to successfully report to CloudWatch Logs.
            The reason for that is, that before it didn't set a task execution role. if you specify none, now it suddenly assumes the role is named ecsTaskExecutionRole, if none is set. If you don't have that role setup, which is completely unnecessary if you work with EC2 and not Fargate, it will fail like this.
            I will try to create this role and update the plugin again, but it is a terrible idea to just do incompatible changes like this. Supporting Fargate and therefore adding Task Execution Role is great, but the default shouldn't just change for the rest of us.
            Also, it is now impossible to even force it to run without task execution role, even if that is what you want.

            Show
            autarchprinceps autarch princeps added a comment - I spoke with the AWS Support and they said the problem was the addition of the attribute restraint ecs.capability.execution-role-awslogs. Previously the task definition only had com.amazonaws.ecs.capability.logging-driver.awslogs, which was sufficient to successfully report to CloudWatch Logs. The reason for that is, that before it didn't set a task execution role. if you specify none, now it suddenly assumes the role is named ecsTaskExecutionRole, if none is set. If you don't have that role setup, which is completely unnecessary if you work with EC2 and not Fargate, it will fail like this. I will try to create this role and update the plugin again, but it is a terrible idea to just do incompatible changes like this. Supporting Fargate and therefore adding Task Execution Role is great, but the default shouldn't just change for the rest of us. Also, it is now impossible to even force it to run without task execution role, even if that is what you want.
            Hide
            autarchprinceps autarch princeps added a comment -

            Even creating the role doesn't make the new task definition work. It still fails at the capabilities.

            Show
            autarchprinceps autarch princeps added a comment - Even creating the role doesn't make the new task definition work. It still fails at the capabilities.
            Hide
            sgraber Stefan Graber added a comment -

            Same here... starting tasks with ecs-plugin 1.18 does not work anymore. Downgrade to 1.17 works.

            Show
            sgraber Stefan Graber added a comment - Same here... starting tasks with ecs-plugin 1.18 does not work anymore. Downgrade to 1.17 works.
            Hide
            webrat Andreas Sieferlinger added a comment -

            Hey, I also have the issue and almost have it traced down with a working workaround but not yet a Fix. Hope i can provide a PR soon.

            Show
            webrat Andreas Sieferlinger added a comment - Hey, I also have the issue and almost have it traced down with a working workaround but not yet a Fix. Hope i can provide a PR soon.
            Show
            pgarbe Philipp Garbe added a comment - See https://github.com/jenkinsci/amazon-ecs-plugin/pull/78

              People

              Assignee:
              pgarbe Philipp Garbe
              Reporter:
              autarchprinceps autarch princeps
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: