Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54428

ECS plugin fails if task definition ARN is used

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: amazon-ecs-plugin
    • Labels:
      None
    • Environment:
      Docker Jenkins LTS 2.138.2 with Amazon ECS Plugin 1.17
      The ECS cloud is configured to use a Task Definition ARN and not to create the task definition for us
    • Similar Issues:
    • Released As:
      v1.18

      Description

      After upgrading the Amazon ECS plugin from 1.16 to 1.17 all ECS slaves fail to start with the exception below:

      Nov 02, 2018 1:55:28 PM hudson.slaves.NodeProvisioner$2 run
      WARNING: Unexpected exception encountered while provisioning agent ECS Slave ecs-agent
      java.lang.NullPointerException
              at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:303)
              at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:278)
              at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud$ProvisioningCallback.call(ECSCloud.java:227)
              at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      Which points to this line: https://github.com/jenkinsci/amazon-ecs-plugin/blob/amazon-ecs-1.17/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java#L303

      Looking at the code and the changelog I suspect that the changes introduced to allow VPC networking support work on the assumption that the task definition template is available but that is not true for us as we pass a pre-configured Task Definition ARN to the cloud configuration.

        Attachments

          Activity

          Hide
          carpnick2 Nick Carpenter added a comment -

          This also might be solved by this PR: https://github.com/jenkinsci/amazon-ecs-plugin/pull/76

          Show
          carpnick2 Nick Carpenter added a comment - This also might be solved by this PR:  https://github.com/jenkinsci/amazon-ecs-plugin/pull/76
          Hide
          pgarbe Philipp Garbe added a comment -

          Can you try it out if v1.18-beta-5 works for you?

          Show
          pgarbe Philipp Garbe added a comment - Can you try it out if v1.18-beta-5 works for you?
          Hide
          stefano_conversocial Stefano Pogliani added a comment -

          Unfortunately after upgrading to 1.18-beta-5 we still see null pointer exceptions that prevent tasks from being created:

          INFO: Started provisioning ECS Slave ecs-agent from ecs-sidekicks with 1 executors. Remaining excess workload: 0
          Nov 13, 2018 1:42:13 PM hudson.slaves.NodeProvisioner$2 run
          INFO: ECS Slave node-8-12-ecs-agent provisioning successfully completed. We have now 2 computer(s)
          Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask
          INFO: [ecs-sidekicks-l47vr]: Starting agent with task definition arn:aws:ecs:xxxxx:xxxxxxxxxxx:task-definition/Node812SidekickTaskDefinition:1}
          Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch
          WARNING: [ecs-sidekicks-l47vr]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ecs-sidekicks-l47vr]
          java.lang.NullPointerException
                  at com.cloudbees.jenkins.plugins.amazonecs.ECSTaskTemplate.isAwsVpcNetworkMode(ECSTaskTemplate.java:353)
                  at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:347)
                  at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:221)
                  at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108)
                  at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
                  at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
                  at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                  at java.lang.Thread.run(Thread.java:748)
          
          Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate
          INFO: [ecs-sidekicks-l47vr]: Terminating ECS Task: null
          
          Show
          stefano_conversocial Stefano Pogliani added a comment - Unfortunately after upgrading to 1.18-beta-5 we still see null pointer exceptions that prevent tasks from being created: INFO: Started provisioning ECS Slave ecs-agent from ecs-sidekicks with 1 executors. Remaining excess workload: 0 Nov 13, 2018 1:42:13 PM hudson.slaves.NodeProvisioner$2 run INFO: ECS Slave node-8-12-ecs-agent provisioning successfully completed. We have now 2 computer(s) Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask INFO: [ecs-sidekicks-l47vr]: Starting agent with task definition arn:aws:ecs:xxxxx:xxxxxxxxxxx:task-definition/Node812SidekickTaskDefinition:1} Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch WARNING: [ecs-sidekicks-l47vr]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ecs-sidekicks-l47vr] java.lang.NullPointerException at com.cloudbees.jenkins.plugins.amazonecs.ECSTaskTemplate.isAwsVpcNetworkMode(ECSTaskTemplate.java:353) at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:347) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:221) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Nov 13, 2018 1:42:13 PM com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate INFO: [ecs-sidekicks-l47vr]: Terminating ECS Task: null
          Hide
          stefano_conversocial Stefano Pogliani added a comment -

          To confirm, the package version:

          Show
          stefano_conversocial Stefano Pogliani added a comment - To confirm, the package version:
          Show
          carpnick2 Nick Carpenter added a comment - Your right Stefano Pogliani .   I think the fix for that line didnt get in until beta-6 –  https://github.com/jenkinsci/amazon-ecs-plugin/blob/amazon-ecs-1.18-beta-6/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java#L347
          Hide
          stefano_conversocial Stefano Pogliani added a comment -

          Oh, sorry Nick Carpenter. I did not realise a beta-6 was available.

          I can confirm that v1.18-beta-6 works correctly.
          Thanks for the fix!

          Show
          stefano_conversocial Stefano Pogliani added a comment - Oh, sorry Nick Carpenter . I did not realise a beta-6 was available. I can confirm that v1.18-beta-6 works correctly. Thanks for the fix!

            People

            Assignee:
            pgarbe Philipp Garbe
            Reporter:
            stefano_conversocial Stefano Pogliani
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: