Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24095

When node is active, reducing number of executors does not reflect reality

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • None

      I have a node with 3 executors.
      There is currently a job running against that node.
      I go to the configuration page for that node and change the number of executors to 1.
      The node now has 2 executors, instead of 1.

      If I do the same procedure on an idle node, it reduces the number of executors to 1 correctly.

          [JENKINS-24095] When node is active, reducing number of executors does not reflect reality

          Daniel Beck added a comment -

          Which executor (number) is the job running on, before and after you reduce the number of executors?

          Daniel Beck added a comment - Which executor (number) is the job running on, before and after you reduce the number of executors?

          Marco Zanchi added a comment -

          Executor 1, before and after.

          Marco Zanchi added a comment - Executor 1, before and after.

          Daniel Beck added a comment -

          Is this the number shown on the UI (1-based) or internal (0-based)?

          Daniel Beck added a comment - Is this the number shown on the UI (1-based) or internal (0-based)?

          Daniel Beck added a comment -

          Please run the following script in script console while the issue occurs:

          println Jenkins.instance.getNode('SlaveName').computer.executors

          Daniel Beck added a comment - Please run the following script in script console while the issue occurs: println Jenkins.instance.getNode( 'SlaveName' ).computer.executors

          Marco Zanchi added a comment -

          Executor doing the current job is 1, UI (1-based).

          I could not run your script, so just did a dump of the node:

          <?xml version="1.0" encoding="UTF-8"?>
          <slave>
          <name>Automated Upgrade</name>
          <description>Buka's computer - UKDT1800</description>
          <remoteFS>c:\JenkinsSlave</remoteFS>
          <numExecutors>1</numExecutors>
          <mode>EXCLUSIVE</mode>
          <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
          <launcher class="hudson.slaves.JNLPLauncher"/>
          <label></label>
          <nodeProperties>
          <hudson.tools.ToolLocationNodeProperty>
          <locations>
          <hudson.tools.ToolLocationNodeProperty_-ToolLocation>
          <type>hudson.tasks.Ant$AntInstallation$DescriptorImpl</type>
          <name>ANT1.7.1</name>
          <home>C:\apache-ant-1.7.1</home>
          </hudson.tools.ToolLocationNodeProperty_-ToolLocation>
          <hudson.tools.ToolLocationNodeProperty_-ToolLocation>
          <type>hudson.tasks.Ant$AntInstallation$DescriptorImpl</type>
          <name>ANT1.7</name>
          <home>C:\jdk1.7.0_25</home>
          </hudson.tools.ToolLocationNodeProperty_-ToolLocation>
          </locations>
          </hudson.tools.ToolLocationNodeProperty>
          </nodeProperties>
          <userId>anonymous</userId>
          </slave>

          Seems to be just a cosmetic issue then. Shall we downgrade this to trivial?

          Marco Zanchi added a comment - Executor doing the current job is 1, UI (1-based). I could not run your script, so just did a dump of the node: <?xml version="1.0" encoding="UTF-8"?> <slave> <name>Automated Upgrade</name> <description>Buka's computer - UKDT1800</description> <remoteFS>c:\JenkinsSlave</remoteFS> <numExecutors>1</numExecutors> <mode>EXCLUSIVE</mode> <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/> <launcher class="hudson.slaves.JNLPLauncher"/> <label></label> <nodeProperties> <hudson.tools.ToolLocationNodeProperty> <locations> <hudson.tools.ToolLocationNodeProperty_-ToolLocation> <type>hudson.tasks.Ant$AntInstallation$DescriptorImpl</type> <name>ANT1.7.1</name> <home>C:\apache-ant-1.7.1</home> </hudson.tools.ToolLocationNodeProperty_-ToolLocation> <hudson.tools.ToolLocationNodeProperty_-ToolLocation> <type>hudson.tasks.Ant$AntInstallation$DescriptorImpl</type> <name>ANT1.7</name> <home>C:\jdk1.7.0_25</home> </hudson.tools.ToolLocationNodeProperty_-ToolLocation> </locations> </hudson.tools.ToolLocationNodeProperty> </nodeProperties> <userId>anonymous</userId> </slave> Seems to be just a cosmetic issue then. Shall we downgrade this to trivial?

          Daniel Beck added a comment - - edited

          Actually it's both not cosmetic (the node will accept a second task) and not a bug (documented to behave this way).

          Two issues:

          • Your API doesn't show anything relevant. The second executor will be removed once the task is finished, numExecutors is already 1 before.
          • I've been unable to reproduce the issue if the task executes on the first executor 'executor #0' (see also /threadDump), only if it's a higher index. Hence the recent questions.

          To clarify, what happens is that every executor has an index from 0...N-1 with N being numExecutors. When reducing the number to M, Jenkins fills up executors 0...M-1, killing all idle ones, and leaving building ones alone. On a five executor node (0...4) with #3 building, setting numExecutors to 2 will result in #0, #1, and #3 existing at first, and #3 removed once the task is finished. This is not reflected on the UI, those indexes are different from the internal ones.

          I'm investigating whether the number of busy executors could be considered when reducing, e.g. reusing the example above, set numExecutors to 2 will result in #0 and #3, and once #3 is finished it'll be killed but a new #1 created.

          Daniel Beck added a comment - - edited Actually it's both not cosmetic (the node will accept a second task) and not a bug (documented to behave this way). Two issues: Your API doesn't show anything relevant. The second executor will be removed once the task is finished, numExecutors is already 1 before. I've been unable to reproduce the issue if the task executes on the first executor 'executor #0' (see also /threadDump), only if it's a higher index. Hence the recent questions. To clarify, what happens is that every executor has an index from 0...N-1 with N being numExecutors. When reducing the number to M, Jenkins fills up executors 0...M-1, killing all idle ones, and leaving building ones alone. On a five executor node (0...4) with #3 building, setting numExecutors to 2 will result in #0, #1, and #3 existing at first, and #3 removed once the task is finished. This is not reflected on the UI, those indexes are different from the internal ones. I'm investigating whether the number of busy executors could be considered when reducing, e.g. reusing the example above, set numExecutors to 2 will result in #0 and #3, and once #3 is finished it'll be killed but a new #1 created.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/Computer.java
          http://jenkins-ci.org/commit/jenkins/05a26da704b9fce80527f5dd6fc32bf57bb2089c
          Log:
          [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing

          When reducing the number of executors from M to N, previously it'd
          kill all idle executors, and then create executors 0...N-1 unless
          they already exist (busy executors).

          This can lead to the situation where e.g. executor #5 is busy and
          retained, and when setting the number of executors to 3, 0...2 are
          created, resulting temporarily having 4 executors.

          This change prevents #2 from being created in the example. When #4
          is finished, it'll be killed anyway and a new executor #2 created,
          so after some time the executor indexes are cleaned up matching
          expectations.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/Computer.java http://jenkins-ci.org/commit/jenkins/05a26da704b9fce80527f5dd6fc32bf57bb2089c Log: [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing When reducing the number of executors from M to N, previously it'd kill all idle executors, and then create executors 0...N-1 unless they already exist (busy executors). This can lead to the situation where e.g. executor #5 is busy and retained, and when setting the number of executors to 3, 0...2 are created, resulting temporarily having 4 executors. This change prevents #2 from being created in the example. When #4 is finished, it'll be killed anyway and a new executor #2 created, so after some time the executor indexes are cleaned up matching expectations.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          core/src/main/java/hudson/model/Computer.java
          http://jenkins-ci.org/commit/jenkins/7fb4591bdcc179f6eddad34175f2e2afe52788aa
          Log:
          Merge pull request #1353 from daniel-beck/JENKINS-24095

          [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing

          Compare: https://github.com/jenkinsci/jenkins/compare/2ea7e628c739...7fb4591bdcc1

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/model/Computer.java http://jenkins-ci.org/commit/jenkins/7fb4591bdcc179f6eddad34175f2e2afe52788aa Log: Merge pull request #1353 from daniel-beck/ JENKINS-24095 [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing Compare: https://github.com/jenkinsci/jenkins/compare/2ea7e628c739...7fb4591bdcc1

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3618
          [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing (Revision 05a26da704b9fce80527f5dd6fc32bf57bb2089c)

          Result = SUCCESS
          daniel-beck : 05a26da704b9fce80527f5dd6fc32bf57bb2089c
          Files :

          • core/src/main/java/hudson/model/Computer.java

          dogfood added a comment - Integrated in jenkins_main_trunk #3618 [FIXED JENKINS-24095] Ensure only numExecutors exist when reducing (Revision 05a26da704b9fce80527f5dd6fc32bf57bb2089c) Result = SUCCESS daniel-beck : 05a26da704b9fce80527f5dd6fc32bf57bb2089c Files : core/src/main/java/hudson/model/Computer.java

            danielbeck Daniel Beck
            inquisitivecrow Marco Zanchi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: