Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55078

No longer able to start ECS slaves

    XMLWordPrintable

Details

    • v1.19

    Description

      We use the ECS plugin to start dynamic slaves when starting jobs. This worked fine, but suddenly stopped working.
      On the UI, all we get is: ‘Jenkins’ doesn’t have label <label>
      The different labels are still in the Jenkins settings however.
      If I look at the log it shows me the following, but it doesn't give any reason or explanation, that would help me understand, what could be causing this.
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure to run task with definition arn:aws:ecs:eu-central-1:x:task-definition/ECS-mgmt:7281 on ECS cluster arn:aws:ecs:eu-central-1:x:cluster/default
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/7b1c9cc6-7eb8-43fc-81cc-5a9d59d09d84
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/8f76d62a-398f-47b0-a883-50f329b41e8c
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/f3e2fddb-17ff-4fe4-9446-eaba4b05b298
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv] hudson.AbortException: Failed to run agent container ECS-wlxkv at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227) at com.cloudbees.jenkins.plugins.amazonecs.ECSLaunche
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch
      WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv]
      hudson.AbortException: Failed to run agent container ECS-wlxkv
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate INFO: [ECS-wlxkv]: Terminating ECS Task: null

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-54893 Error provisioning slaves with ECS plugin 1.18

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            autarchprinceps autarch princeps added a comment -

            I was able to fix the issue for me, by downgrading the plugin, so that seems to indicate a bug of some kind

            autarchprinceps autarch princeps added a comment - I was able to fix the issue for me, by downgrading the plugin, so that seems to indicate a bug of some kind
            autarchprinceps autarch princeps added a comment -

            I spoke with the AWS Support and they said the problem was the addition of the attribute restraint ecs.capability.execution-role-awslogs. Previously the task definition only had com.amazonaws.ecs.capability.logging-driver.awslogs, which was sufficient to successfully report to CloudWatch Logs.
            The reason for that is, that before it didn't set a task execution role. if you specify none, now it suddenly assumes the role is named ecsTaskExecutionRole, if none is set. If you don't have that role setup, which is completely unnecessary if you work with EC2 and not Fargate, it will fail like this.
            I will try to create this role and update the plugin again, but it is a terrible idea to just do incompatible changes like this. Supporting Fargate and therefore adding Task Execution Role is great, but the default shouldn't just change for the rest of us.
            Also, it is now impossible to even force it to run without task execution role, even if that is what you want.

            autarchprinceps autarch princeps added a comment - I spoke with the AWS Support and they said the problem was the addition of the attribute restraint ecs.capability.execution-role-awslogs. Previously the task definition only had com.amazonaws.ecs.capability.logging-driver.awslogs, which was sufficient to successfully report to CloudWatch Logs. The reason for that is, that before it didn't set a task execution role. if you specify none, now it suddenly assumes the role is named ecsTaskExecutionRole, if none is set. If you don't have that role setup, which is completely unnecessary if you work with EC2 and not Fargate, it will fail like this. I will try to create this role and update the plugin again, but it is a terrible idea to just do incompatible changes like this. Supporting Fargate and therefore adding Task Execution Role is great, but the default shouldn't just change for the rest of us. Also, it is now impossible to even force it to run without task execution role, even if that is what you want.
            autarchprinceps autarch princeps added a comment -

            Even creating the role doesn't make the new task definition work. It still fails at the capabilities.

            autarchprinceps autarch princeps added a comment - Even creating the role doesn't make the new task definition work. It still fails at the capabilities.
            sgraber Stefan Graber added a comment -

            Same here... starting tasks with ecs-plugin 1.18 does not work anymore. Downgrade to 1.17 works.

            sgraber Stefan Graber added a comment - Same here... starting tasks with ecs-plugin 1.18 does not work anymore. Downgrade to 1.17 works.
            webrat Andreas Sieferlinger added a comment -

            Hey, I also have the issue and almost have it traced down with a working workaround but not yet a Fix. Hope i can provide a PR soon.

            webrat Andreas Sieferlinger added a comment - Hey, I also have the issue and almost have it traced down with a working workaround but not yet a Fix. Hope i can provide a PR soon.
            pgarbe Philipp Garbe added a comment -

            See https://github.com/jenkinsci/amazon-ecs-plugin/pull/78

            pgarbe Philipp Garbe added a comment - See https://github.com/jenkinsci/amazon-ecs-plugin/pull/78

            People

              pgarbe Philipp Garbe
              autarchprinceps autarch princeps
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: