Still an issue with Jenkins 2.60.3 and plugin version 1.11.
When launching a bunch of parallel jobs, I see the plugin starts launching agents as supposed but as there are still jobs in queue after the first jobs finish the agents stay on the list as offline even though the container tasks are stopped and containers deleted as supposed from ECS cluster. And as the plugin thinks the offline agents use all available ECS cpu capacity, no new agents are launched thus the rest of the jobs do not get run. This is a blocking bug for us, the plugin is virtually unusable as we would need to constantly manually delete the offline agents.
Mika Karjalainen
added a comment - Still an issue with Jenkins 2.60.3 and plugin version 1.11.
When launching a bunch of parallel jobs, I see the plugin starts launching agents as supposed but as there are still jobs in queue after the first jobs finish the agents stay on the list as offline even though the container tasks are stopped and containers deleted as supposed from ECS cluster. And as the plugin thinks the offline agents use all available ECS cpu capacity, no new agents are launched thus the rest of the jobs do not get run. This is a blocking bug for us, the plugin is virtually unusable as we would need to constantly manually delete the offline agents.
I too am seeing similar, with build executors gradually accumulating in an offline state. Jobs to continue to execute okay, but this accumulation of offline executors is problematic, especially on a busy server.
Jenkins version 2.121 (based on the official jenkins/jenkins:alpine Docker image) with plugin version 1.14, JNLP slaves using the jenkinsci/jnlp-slave Docker image.
WARNING: jenkins.util.Timer [#3] for ci-jenkins-build-executors-42b6bb8c663f terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746) at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:713) at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:88) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:227) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1551) at jenkins.model.Nodes$6.run(Nodes.java:261) at hudson.model.Queue._withLock(Queue.java:1378) at hudson.model.Queue.withLock(Queue.java:1255) at jenkins.model.Nodes.removeNode(Nodes.java:252) at jenkins.model.Jenkins.removeNode(Jenkins.java:2065) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:70) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave$1.check(ECSSlave.java:82) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave$1.check(ECSSlave.java:70) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Queue.java:1378) at hudson.model.Queue.withLock(Queue.java:1255) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
WARNING: Computer.threadPoolForRemoting [#8281] for ci-jenkins-build-executors-429623dffe91 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:99) at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:664) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Nicola Worthington
added a comment - - edited I too am seeing similar, with build executors gradually accumulating in an offline state. Jobs to continue to execute okay, but this accumulation of offline executors is problematic, especially on a busy server.
Jenkins version 2.121 (based on the official jenkins/jenkins:alpine Docker image) with plugin version 1.14, JNLP slaves using the jenkinsci/jnlp-slave Docker image.
WARNING: jenkins.util.Timer [#3] for ci-jenkins-build-executors-42b6bb8c663f terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746) at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:713) at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:88) at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:227) at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1551) at jenkins.model.Nodes$6.run(Nodes.java:261) at hudson.model.Queue._withLock(Queue.java:1378) at hudson.model.Queue.withLock(Queue.java:1255) at jenkins.model.Nodes.removeNode(Nodes.java:252) at jenkins.model.Jenkins.removeNode(Jenkins.java:2065) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:70) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave$1.check(ECSSlave.java:82) at com.cloudbees.jenkins.plugins.amazonecs.ECSSlave$1.check(ECSSlave.java:70) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Queue.java:1378) at hudson.model.Queue.withLock(Queue.java:1255) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
WARNING: Computer.threadPoolForRemoting [#8281] for ci-jenkins-build-executors-429623dffe91 terminated java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:99) at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:664) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
Seeing this issue occur when we've more than 30 agents connected, it's only effecting ECS tasks EC2 seems to spawn fine. What I've noted is that only some of the ECS agent 'types' we have fail other ECS agent types seem to launch and connect fine with the same underlying ECS cluster.
here's a cleanup script that may help for those with busy servers.
Damien Roche
added a comment - Seeing this issue occur when we've more than 30 agents connected, it's only effecting ECS tasks EC2 seems to spawn fine. What I've noted is that only some of the ECS agent 'types' we have fail other ECS agent types seem to launch and connect fine with the same underlying ECS cluster.
here's a cleanup script that may help for those with busy servers.
String agentList = ""
String agentPrefix = "example"
Integer agentTotal = 0
Integer ofType = 0
Integer ofDeleted = 0
// Itterate nodes
Jenkins.instance.nodes.each {
//println "Checking agent: $it.nodeName"
if (it.nodeName.contains(agentPrefix)){
//println it.computer.offlineCause.toString()
if (it.computer.offlineCause.toString().contains( 'Time out for last 5 try ' )) {
agentList += it.nodeName + "\n"
it.computer.doDoDelete()
ofDeleted +=1
}
ofType +=1
}
agentTotal +=1
}
println "Deleted Agent list: \n" + agentList
println "Total: ${agentTotal} Total type: ${ofType} Deleted: ${ofDeleted}"
So there are a couple of different issues where they don't get cleaned up automatically.
INFO: Created Slave: cidd-9bd0fafd9be3
INFO: Running task definition arn:aws:ecs:us-east-1:123456789012:task-definition/cidd-t2-small-generic:1 on slave cidd-9bd0fafd9be3
INFO: Slave cidd-9bd0fafd9be3 - Slave Task Started : arn:aws:ecs:us-east-1:123456789012:task/example-omited
INFO: ECS Slave cidd-9bd0fafd9be3 (ecs task arn:aws:ecs:us-east-1:123456789012:task/example-omited) connected
WARNING: Computer.threadPoolForRemoting [#1754] for cidd-9bd0fafd9be3 terminated
WARNING: Making cidd-9bd0fafd9be3 offline because it’s not responding
Even tho it was terminated the node/agent remained in the list
INFO: Created Slave: cidd-9b44dbcdddde
INFO: Running task definition arn:aws:ecs:us-east-1:123456789012:task-definition/cidd-t2-small-generic:1 on slave cidd-9b44dbcdddde
INFO: Slave cidd-9b44dbcdddde - Slave Task Started : arn:aws:ecs:us-east-1:123456789012:task/example-omitted
INFO: ECS Slave cidd-9b44dbcdddde (ecs task arn:aws:ecs:us-east-1:123456789012:task/example-omitted) connected
WARNING: Making cidd-9b44dbcdddde offline temporarily due to the use of an old slave.jar
WARNING: Computer.threadPoolForRemoting [#1753] for cidd-9b44dbcdddde terminated
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
Similar issue node/agent was not removed from the list. But I believe at least this node is using a newer version of remoting than the master.
Will downgrade this to match and try again.
Damien Roche
added a comment - So there are a couple of different issues where they don't get cleaned up automatically.
INFO: Created Slave: cidd-9bd0fafd9be3
INFO: Running task definition arn:aws:ecs:us-east-1:123456789012:task-definition/cidd-t2-small- generic :1 on slave cidd-9bd0fafd9be3
INFO: Slave cidd-9bd0fafd9be3 - Slave Task Started : arn:aws:ecs:us-east-1:123456789012:task/example-omited
INFO: ECS Slave cidd-9bd0fafd9be3 (ecs task arn:aws:ecs:us-east-1:123456789012:task/example-omited) connected
WARNING: Computer.threadPoolForRemoting [#1754] for cidd-9bd0fafd9be3 terminated
WARNING: Making cidd-9bd0fafd9be3 offline because it’s not responding
Even tho it was terminated the node/agent remained in the list
INFO: Created Slave: cidd-9b44dbcdddde
INFO: Running task definition arn:aws:ecs:us-east-1:123456789012:task-definition/cidd-t2-small- generic :1 on slave cidd-9b44dbcdddde
INFO: Slave cidd-9b44dbcdddde - Slave Task Started : arn:aws:ecs:us-east-1:123456789012:task/example-omitted
INFO: ECS Slave cidd-9b44dbcdddde (ecs task arn:aws:ecs:us-east-1:123456789012:task/example-omitted) connected
WARNING: Making cidd-9b44dbcdddde offline temporarily due to the use of an old slave.jar
WARNING: Computer.threadPoolForRemoting [#1753] for cidd-9b44dbcdddde terminated
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
WARNING: Making cidd-9b44dbcdddde offline because it’s not responding
Similar issue node/agent was not removed from the list. But I believe at least this node is using a newer version of remoting than the master.
Will downgrade this to match and try again.
Looks to have resolved it on our side. Much appreciated.
Damien Roche
added a comment - Looks to have resolved it on our side. Much appreciated.
Philipp Garbe
Eric Goedtel
Votes:
6Vote for this issue
Watchers:
13Start watching this issue
Created:
Updated:
Resolved:
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1740202179662},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1740202179663},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1740202179669},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":true},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1740202179673},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bonfire Browser","id":"customfield_10229","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Operating System","id":"customfield_10231","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire Screen Resolution","id":"customfield_10244","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire URL","id":"customfield_10237","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire User Agent","id":"customfield_10226","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Bonfire jQuery Version","id":"customfield_10252","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Business Value","id":"customfield_10333","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10720","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic Color","id":"customfield_10328","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10325","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10327","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10326","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10331","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_10330","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"GitHub Users to Authorize as Committers","id":"customfield_10323","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Issue Tracker","id":"customfield_11320","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10228","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Browser","id":"customfield_10230","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Document Mode","id":"customfield_10258","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10232","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10233","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10234","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10236","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10238","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Operating System","id":"customfield_10239","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10245","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10246","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10247","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10248","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10249","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture Screen Resolution","id":"customfield_10250","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10240","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10241","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10242","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture URL","id":"customfield_10243","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10225","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture User Agent","id":"customfield_10227","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10251","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10253","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10254","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10255","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10256","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"JIRA Capture jQuery Version","id":"customfield_10257","key":"com.atlassian.bonfire.plugin:bonfire-text","isShown":true},{"name":"Meeting minutes URL","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":false},{"name":"New Repository Name","id":"customfield_10321","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Original story points","id":"customfield_11423","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11420","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Plugin Description","id":"customfield_10322","key":"com.atlassian.jira.plugin.system.customfieldtypes:textarea","isShown":true},{"name":"Raised During","id":"customfield_10220","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Raised During","id":"customfield_10221","key":"com.atlassian.bonfire.plugin:bonfire-session-cft","isShown":false},{"name":"Rank","id":"customfield_10324","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Released As","id":"customfield_10620","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Repository URL","id":"customfield_10320","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true},{"name":"Similar Issues","id":"customfield_10520","key":"com.suggestimate:similar-issues-custom-field","isShown":true},{"name":"Sprint","id":"customfield_10329","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10332","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Team","id":"customfield_11424","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Test Sessions","id":"customfield_10222","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10223","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Test Sessions","id":"customfield_10224","key":"com.atlassian.bonfire.plugin:bonfire-multi-session-cft","isShown":false},{"name":"Testing Status","id":"customfield_10259","key":"com.atlassian.bonfire.plugin:bonfire-testing-status-cft","isShown":false},{"name":"URL","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:url","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Target end","id":"customfield_11422","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11421","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1740202179669},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14673&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10001\"\n title=\"Epic\"\n value=\"10001\">Epic</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14680&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14681&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14670&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Patch\"\n value=\"5\">Patch</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14685&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10002\"\n title=\"Story\"\n value=\"10002\">Story</option>\n \n <option class=\" imagebacked 10730 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=14688&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=10172&size=small\"\n title=\"Jenkins\"\n value=\"10172\">\n Jenkins (JENKINS)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10050&size=small\"\n title=\"test\"\n value=\"10050\">\n test (TEST)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://issues.jenkins.io/secure/useravatar?size=xsmall&avatarId=10293\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-88 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/needinfo.png\" value=\"10001\" title=\"Untriaged\" data-simple-status=\"{"id":"10001","name":"Untriaged","description":"New issue sits in this state until a developer agrees that this is a security issue","iconUrl":"/images/icons/statuses/needinfo.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Untriaged</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10005\" title=\"In Review\" data-simple-status=\"{"id":"10005","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"10002\" title=\"Fix Prepared\" data-simple-status=\"{"id":"10002","name":"Fix Prepared","description":"A fix is implemented and is waiting for the next security release","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Fix Prepared</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Verified\" data-simple-status=\"{"id":"10000","name":"Verified","description":"Verified","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10203\" title=\"Fixed but Unreleased\" data-simple-status=\"{"id":"10203","name":"Fixed but Unreleased","description":"This change has been implemented and merged, but not yet released.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fixed but Unreleased</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A developer had implemented a fix and is waiting for a feedback from the reporter.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10003\" title=\"To Do\" data-simple-status=\"{"id":"10003","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10004\" title=\"Done\" data-simple-status=\"{"id":"10004","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n slave\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"slave\" title=\"slave\" selected=\"selected\">slave</option>\n </select>\n </div>\n \n","jql":"labels = slave","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Still an issue with Jenkins 2.60.3 and plugin version 1.11.
When launching a bunch of parallel jobs, I see the plugin starts launching agents as supposed but as there are still jobs in queue after the first jobs finish the agents stay on the list as offline even though the container tasks are stopped and containers deleted as supposed from ECS cluster. And as the plugin thinks the offline agents use all available ECS cpu capacity, no new agents are launched thus the rest of the jobs do not get run. This is a blocking bug for us, the plugin is virtually unusable as we would need to constantly manually delete the offline agents.