Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55078

No longer able to start ECS slaves

    XMLWordPrintable

Details

    • v1.19

    Description

      We use the ECS plugin to start dynamic slaves when starting jobs. This worked fine, but suddenly stopped working.
      On the UI, all we get is: ‘Jenkins’ doesn’t have label <label>
      The different labels are still in the Jenkins settings however.
      If I look at the log it shows me the following, but it doesn't give any reason or explanation, that would help me understand, what could be causing this.
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure to run task with definition arn:aws:ecs:eu-central-1:x:task-definition/ECS-mgmt:7281 on ECS cluster arn:aws:ecs:eu-central-1:x:cluster/default
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/7b1c9cc6-7eb8-43fc-81cc-5a9d59d09d84
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/8f76d62a-398f-47b0-a883-50f329b41e8c
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask WARNING: [ECS-wlxkv]: Failure reason=ATTRIBUTE, arn=arn:aws:ecs:eu-central-1:x:container-instance/f3e2fddb-17ff-4fe4-9446-eaba4b05b298
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv] hudson.AbortException: Failed to run agent container ECS-wlxkv at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227) at com.cloudbees.jenkins.plugins.amazonecs.ECSLaunche
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch
      WARNING: [ECS-wlxkv]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[ECS-wlxkv]
      hudson.AbortException: Failed to run agent container ECS-wlxkv
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:227)
      at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:108)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Dec 07, 2018 1:02:29 PM com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate INFO: [ECS-wlxkv]: Terminating ECS Task: null

      Attachments

        Issue Links

          Activity

            autarchprinceps autarch princeps created issue -
            autarchprinceps autarch princeps made changes -
            Field Original Value New Value
            Comment [ This is what I think the slave log says in correspondence:
            13:07:48
            Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior

            13:07:50
            Dec 07, 2018 1:07:50 PM hudson.remoting.jnlp.Main createEngine

            13:07:50
            INFO: Setting up agent: ECS-ea9fff4b4f18

            13:07:50
            Dec 07, 2018 1:07:50 PM hudson.remoting.jnlp.Main$CuiListener <init>

            13:07:50
            INFO: Jenkins agent is running in headless mode.

            13:07:50
            Dec 07, 2018 1:07:50 PM hudson.remoting.Engine startEngine

            13:07:50
            INFO: Using Remoting version: 3.27

            13:07:50
            Dec 07, 2018 1:07:50 PM hudson.remoting.Engine startEngine

            13:07:50
            WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars

            13:07:50
            Dec 07, 2018 1:07:50 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:50
            INFO: Locating server among [https://jenkins.prod.mgmt.aws-x.com/]

            13:07:52
            Dec 07, 2018 1:07:52 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve

            13:07:52
            INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]

            13:07:52
            Dec 07, 2018 1:07:52 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:52
            INFO: Agent discovery successful

            13:07:52
            Agent address: jenkins.prod.mgmt.aws-x.com

            13:07:52
            Agent port: 50000

            13:07:52
            Identity: 65:c7:02:f6:43:aa:38:15:a0:6d:f6:79:89:64:62:4c

            13:07:52
            Dec 07, 2018 1:07:52 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:52
            INFO: Handshaking

            13:07:52
            Dec 07, 2018 1:07:52 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:52
            INFO: Connecting to jenkins.prod.mgmt.aws-x.com:50000

            13:07:52
            Dec 07, 2018 1:07:52 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:52
            INFO: Trying protocol: JNLP4-connect

            13:07:53
            Dec 07, 2018 1:07:53 PM hudson.remoting.jnlp.Main$CuiListener status

            13:07:53
            INFO: Remote identity confirmed: 65:c7:02:f6:43:aa:38:15:a0:6d:f6:79:89:64:62:4c

            13:08:02
            Dec 07, 2018 1:08:02 PM hudson.remoting.jnlp.Main$CuiListener status

            13:08:02
            INFO: Connected

            13:08:26
            Dec 07, 2018 1:08:26 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

            13:08:26
            WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.envinject.EnvInjectComputerListener$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/

            13:08:27
            Dec 07, 2018 1:08:27 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

            13:08:27
            WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.gitclient.Git$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/

            13:08:47
            Dec 07, 2018 1:08:47 PM org.jenkinsci.remoting.util.AnonymousClassWarnings warn

            13:08:47
            WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ ]
            pgarbe Philipp Garbe made changes -
            Assignee Jan Roehrich [ roehrijn2 ] Philipp Garbe [ pgarbe ]
            pgarbe Philipp Garbe made changes -
            Released As v1.19
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            pgarbe Philipp Garbe made changes -
            Link This issue is duplicated by JENKINS-54893 [ JENKINS-54893 ]
            pgarbe Philipp Garbe made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            pgarbe Philipp Garbe made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              pgarbe Philipp Garbe
              autarchprinceps autarch princeps
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: