Major We have multiple slave nodes, on windows, all having the number of executors set to 1.
Our workflow is as follows:
Job 1 requests an executor with a specific label and calls Job 2 (waiting for it's completion)
Job 2 requests another executor with same label and calls Job 3 (waiting for it's completion).
Very seldom, we have runs where the same node (with the number of executors set to 1) gets assigned 2 times (we always assign 2 executors to a run).
[JENKINS-24831] Executor gets assigned multiple times
Octavian Ciuhandu
added a comment - I'm attaching 2 screenshots with the definitions of Job 1 and Job 2.
As for plugins, we use: Gearman 0.0.7, Build timeout 1.14, Event Publisher 0.0.3, Node and Label parameter 1.5.1, Parameterized Trigger 2.25, Throttle Concurrent Builds 1.8.3
Octavian Ciuhandu
added a comment - I'm attaching 2 screenshots with the definitions of Job 1 and Job 2.
As for plugins, we use: Gearman 0.0.7, Build timeout 1.14, Event Publisher 0.0.3, Node and Label parameter 1.5.1, Parameterized Trigger 2.25, Throttle Concurrent Builds 1.8.3 Octavian Ciuhandu
added a comment - Job 01 Octavian Ciuhandu
added a comment - Job 02 Please explain what you mean here:
gets assigned 2 times (we always assign 2 executors to a run).
The jobs' config.xml would also be helpful.
Why are you passing in a parameter to 02 if it doesn't take parameters?
Octavian Ciuhandu
added a comment - Job 2 calls another job (Job 3) that actually uses both parameters.
Job 1 config.xml: http://paste.openstack.org/show/114710/
Job 2 config.xml: http://paste.openstack.org/show/114711/
By "gets assigned 2 times" I mean that when I check the 2 parameters (that are set in Job 1 and Job 2) they have the same value, e.g.:
hyperv01: node1.domain.tld
hyperv02: node1.domain.tld
Since each job states that the parameter hyperv01/hyperv02 takes value $NODE_NAME and there's only one executor per slave node, this should be impossible.
Octavian Ciuhandu
added a comment - Job 2 calls another job (Job 3) that actually uses both parameters.
Job 1 config.xml: http://paste.openstack.org/show/114710/
Job 2 config.xml: http://paste.openstack.org/show/114711/
By "gets assigned 2 times" I mean that when I check the 2 parameters (that are set in Job 1 and Job 2) they have the same value, e.g.:
hyperv01: node1.domain.tld
hyperv02: node1.domain.tld
Since each job states that the parameter hyperv01/hyperv02 takes value $NODE_NAME and there's only one executor per slave node, this should be impossible. Octavian Ciuhandu
added a comment - As a note, it's something that happens only occasionally, we have seen this happening about around 5 times in the last week or so, and we have not been able to identify any possible reason for this.
Octavian Ciuhandu
added a comment - As a note, it's something that happens only occasionally, we have seen this happening about around 5 times in the last week or so, and we have not been able to identify any possible reason for this. I have exactly the same issue with Amazon EC2 Container Service plugin. Jenkins creates new ECS task with labeled slave that has only 1 executor and starts build there, then another job comes that uses the same label. For some reason it assigns new build to the same only executor. Once one of the builds finishes slave is killed and second build fails because of it. MatrixJob creates second build on executor
FATAL: command execution failed java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:311) at hudson.remoting.Channel.close(Channel.java:1295) at hudson.remoting.Channel.close(Channel.java:1263) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:708) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:96) at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:626) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused: java.io.IOException: Backing channel 'JNLP4-connect connection from ip-172-29-16-239.eu-west-1.compute.internal/172.29.16.239:9050' is disconnected. at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy100.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1043) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1035) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:735) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490) at hudson.model.Run.execute(Run.java:1735) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Import campaign adwords_id= '239459797' Build step 'Execute shell' marked build as failure FATAL: null java.lang.NullPointerException at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir.tempDir(UnbindableDir.java:67) at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir.secretsDir(UnbindableDir.java:62) at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir.access$000(UnbindableDir.java:23) at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84) at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113) at hudson.model.Build$BuildExecution.doRun(Build.java:174) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490) at hudson.model.Run.execute(Run.java:1735) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:405) Finished: FAILURE
Please see screenshot below.
Jenkins version: 2.60.1
Jnlp slave: https://hub.docker.com/r/jenkinsci/jnlp-slave/ latest
AWS ECS plugin: 1.11
Katsuya Tomioka
added a comment - I'm seeing the same issue with the nomad plugin 0.4, Jenkins 2.121.2. If two builds are started close each other (while waiting for executors to be provisioned) both jobs are assigned to the same executor.
Katsuya Tomioka
added a comment - I'm seeing the same issue with the nomad plugin 0.4, Jenkins 2.121.2. If two builds are started close each other (while waiting for executors to be provisioned) both jobs are assigned to the same executor. 
Stacey Fletcher
added a comment - Was the root cause ever figured out here? I am also seeing the same thing in our Jenkins.
Stacey Fletcher
added a comment - Was the root cause ever figured out here? I am also seeing the same thing in our Jenkins. 
Could you provide a minimal, complete, verifiable example of jobs that have this problem? Which plugins are used (e.g. to trigger + wait, to require two executors per run, ...), in which versions?