Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54893

Error provisioning slaves with ECS plugin 1.18

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • amazon-ecs-plugin
    • None
    • amazon-ecs 1.18
      jenkins 2.38.3

    Description

      After upgrading the Jenkins ECS plugin to 1.18, slaves no longer provision properly.

      I receive the following error:

      Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ecs-fargate-d3jvq]

      java.lang.NullPointerException at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.getDockerRunCommand(ECSLauncher.java:247)

       

      Additional logging:

      Nov 27, 2018 6:49:05 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplateNov 27, 2018 6:49:05 PM com.cloudbees.jenkins.plugins.amazonecs.ECSService registerTemplateINFO: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-west-2:XXX:task-definition/ecs-fargate-jenkins-node-8:3385,ContainerDefinitions: [{Name: ecs-fargate-jenkins-node-8,Image: 202306134469.dkr.ecr.us-west-2.amazonaws.com/ookla-jnlp-nodejs-8,Cpu: 2048,Memory: 4096,Links: [],PortMappings: [],Essential: true,EntryPoint: [],Command: [],Environment: [],MountPoints: [],VolumesFrom: [],User: jenkins,Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],LogConfiguration: {LogDriver: awslogs,Options: {awslogs-group=fargate, awslogs-region=us-west-2, awslogs-stream-prefix=node-8}},}],Family: ecs-fargate-jenkins-node-8,ExecutionRoleArn: arn:aws:iam::XXX:role/ecsTaskExecutionRole,NetworkMode: awsvpc,Revision: 3385,Volumes: [],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.docker-remote-api.1.17,}, {Name: ecs.capability.execution-role-ecr-pull,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}, {Name: ecs.capability.task-eni,}, {Name: com.amazonaws.ecs.capability.ecr-auth,}, {Name: ecs.capability.execution-role-awslogs,}, {Name: com.amazonaws.ecs.capability.logging-driver.awslogs,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.19,}],PlacementConstraints: [],Compatibilities: [EC2, FARGATE],RequiresCompatibilities: [FARGATE],Cpu: 2048,Memory: 4096}Nov 27, 2018 6:49:05 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTaskINFO: [ecs-fargate-d3jvq]: Starting agent with task definition arn:aws:ecs:us-west-2:XXX:task-definition/ecs-fargate-jenkins-node-8:3385}Nov 27, 2018 6:49:05 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchWARNING: [ecs-fargate-d3jvq]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ecs-fargate-d3jvq]java.lang.NullPointerException at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.getDockerRunCommand(ECSLauncher.java:247) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:220) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
      

      Downgrading to 1.17 resolves the issue. These are FARGATE ECS tasks (not EC2)

       

      Attachments

        Issue Links

          Activity

            jaredrsowers Jared Sowers added a comment -

            We are having the same issue using EC2 and not Fargate. Downgrading to 1.17 fixed the issue.

             

            This only happens when there are multiple builds trying to spin up instances, if i cancel all but 1 pending build then it can successfully create the container, but if there are multiple it fails with this error.

            jaredrsowers Jared Sowers added a comment - We are having the same issue using EC2 and not Fargate. Downgrading to 1.17 fixed the issue.   This only happens when there are multiple builds trying to spin up instances, if i cancel all but 1 pending build then it can successfully create the container, but if there are multiple it fails with this error.

            It doesn't seem obvious to me that this is a duplicate of JENKINS-55078 as the errors are different, but it's possible that the fix for 55078 also fixes this. When 1.19 is released, I'll test and report back.

            Thanks!

            mignaulo Olivier Mignault added a comment - It doesn't seem obvious to me that this is a duplicate of  JENKINS-55078  as the errors are different, but it's possible that the fix for 55078 also fixes this. When 1.19 is released, I'll test and report back. Thanks!
            pgarbe Philipp Garbe added a comment -

            Yes, it's the same fix, that's why I closed this one. You can try out the 1.19-beta-1 already, if you want. 

            pgarbe Philipp Garbe added a comment - Yes, it's the same fix, that's why I closed this one. You can try out the 1.19-beta-1 already, if you want. 

            Thank you pgarbe, I can confirm 1.19 fixed our issues.

            mignaulo Olivier Mignault added a comment - Thank you pgarbe , I can confirm 1.19 fixed our issues.

            People

              pgarbe Philipp Garbe
              mignaulo Olivier Mignault
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: