Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7291

Flyweight jobs and zero executors

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      In a cloud-enabled environment where Hudson has zero static executors, there will be no Computer object until a node is provisioned, so even a fly-weight job would wait for an allocation of a node.

      (And IIRC, I also saw that the flyweight job still used up a regular executor.)

      Needs more analysis.

        Attachments

          Issue Links

            Activity

            kohsuke Kohsuke Kawaguchi created issue -
            Hide
            recampbell Ryan Campbell added a comment -

            It would be nice if one could allow flyweight tasks on master even when no executors are configured.

            Show
            recampbell Ryan Campbell added a comment - It would be nice if one could allow flyweight tasks on master even when no executors are configured.
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Assignee Jesse Glick [ jglick ]
            Hide
            jglick Jesse Glick added a comment -

            Using nectar-vmware on an instance with zero master executors and a cloud configured to provide one executor in a cloud slave on demand, I made a matrix job with a slave axis with one value corresponding to the cloud label. When I asked to start it, the cloud slave was in fact spun up, but the one executor was consumed by the matrix build (with nothing left over for the configuration build):

            Building remotely on VM5-stuff-ubuntu1 in workspace …
            Triggering vmware
            Configuration vmware is still in the queue: Waiting for next available executor on VM5-stuff-ubuntu1
            …later…
            Configuration vmware is still in the queue: VM5-stuff-ubuntu1 is offline
            Aborted
            Cancelled vmware
            
            Show
            jglick Jesse Glick added a comment - Using nectar-vmware on an instance with zero master executors and a cloud configured to provide one executor in a cloud slave on demand, I made a matrix job with a slave axis with one value corresponding to the cloud label. When I asked to start it, the cloud slave was in fact spun up, but the one executor was consumed by the matrix build (with nothing left over for the configuration build): Building remotely on VM5-stuff-ubuntu1 in workspace … Triggering vmware Configuration vmware is still in the queue: Waiting for next available executor on VM5-stuff-ubuntu1 …later… Configuration vmware is still in the queue: VM5-stuff-ubuntu1 is offline Aborted Cancelled vmware
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-17276 [ JENKINS-17276 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/Queue.java
            http://jenkins-ci.org/commit/jenkins/7e74ab6a6479ec1ee7564663d080f276996930c5
            Log:
            [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors.


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/7e74ab6a6479ec1ee7564663d080f276996930c5 Log: [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors. – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Status Reopened [ 4 ] Open [ 1 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractCIBase.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/model/Queue.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/model/Jenkins.java
            core/src/main/resources/lib/hudson/executors.jelly
            http://jenkins-ci.org/commit/jenkins/2c5b57fcc1f39ed39057254e802f4183db5aa0dc
            Log:
            Always adding Computer for master as a fallback

            The original proposed fix for JENKINS-7291 creates a Computer object
            transitively. This seems unwise as it violates the design of Computer
            as stated in the javadoc, and for example we can end up creating two
            Computers for the master.

            I think a better fix is to create a Computer for the master all the
            time, even if there's no executors configured. The discrimination in
            Queue.makeBuildable would ensure that such phantom Computer is only used
            as a last resort.

            I've also tweaked executors.jelly a bit. I simplified it somewhat
            based on the idea that "if there's only one computer to show, the
            context is likely making it obvious".

            (I must be missing the intricacy in the current code.)


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractCIBase.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Queue.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/model/Jenkins.java core/src/main/resources/lib/hudson/executors.jelly http://jenkins-ci.org/commit/jenkins/2c5b57fcc1f39ed39057254e802f4183db5aa0dc Log: Always adding Computer for master as a fallback The original proposed fix for JENKINS-7291 creates a Computer object transitively. This seems unwise as it violates the design of Computer as stated in the javadoc, and for example we can end up creating two Computers for the master. I think a better fix is to create a Computer for the master all the time, even if there's no executors configured. The discrimination in Queue.makeBuildable would ensure that such phantom Computer is only used as a last resort. I've also tweaked executors.jelly a bit. I simplified it somewhat based on the idea that "if there's only one computer to show, the context is likely making it obvious". (I must be missing the intricacy in the current code.) – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/model/AbstractCIBase.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/model/Queue.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/model/Jenkins.java
            core/src/main/resources/lib/hudson/executors.jelly
            test/src/test/java/hudson/model/QueueTest.java
            http://jenkins-ci.org/commit/jenkins/a114c693bc61c564232e78c9fffb5de1ef946ea8
            Log:
            [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors.

            Always adding Computer for master as a fallback

            The original proposed fix for JENKINS-7291 creates a Computer object
            transitively. This seems unwise as it violates the design of Computer
            as stated in the javadoc, and for example we can end up creating two
            Computers for the master.

            I think a better fix is to create a Computer for the master all the
            time, even if there's no executors configured. The discrimination in
            Queue.makeBuildable would ensure that such phantom Computer is only used
            as a last resort (statistically speaking).

            I've also tweaked executors.jelly a bit. I simplified it somewhat
            based on the idea that "if there's only one computer to show, the
            context is likely making it obvious".

            (I must be missing the intricacy in the current code.)

            Originally developed in a branch at
            2c5b57fcc1f39ed39057254e802f4183db5aa0dc then squashed for clarity.


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/model/AbstractCIBase.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Queue.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/model/Jenkins.java core/src/main/resources/lib/hudson/executors.jelly test/src/test/java/hudson/model/QueueTest.java http://jenkins-ci.org/commit/jenkins/a114c693bc61c564232e78c9fffb5de1ef946ea8 Log: [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors. Always adding Computer for master as a fallback The original proposed fix for JENKINS-7291 creates a Computer object transitively. This seems unwise as it violates the design of Computer as stated in the javadoc, and for example we can end up creating two Computers for the master. I think a better fix is to create a Computer for the master all the time, even if there's no executors configured. The discrimination in Queue.makeBuildable would ensure that such phantom Computer is only used as a last resort (statistically speaking). I've also tweaked executors.jelly a bit. I simplified it somewhat based on the idea that "if there's only one computer to show, the context is likely making it obvious". (I must be missing the intricacy in the current code.) Originally developed in a branch at 2c5b57fcc1f39ed39057254e802f4183db5aa0dc then squashed for clarity. – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2410

            Result = SUCCESS

            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2410 Result = SUCCESS
            Hide
            cdemarey Christophe Demarey added a comment -

            Does it mean that a workspace will be created on the master?

            Show
            cdemarey Christophe Demarey added a comment - Does it mean that a workspace will be created on the master?
            Hide
            jglick Jesse Glick added a comment -

            @cdemarey: I do not think the matrix job flyweight task creates a workspace. If it did, it would probably be empty.

            Show
            jglick Jesse Glick added a comment - @cdemarey: I do not think the matrix job flyweight task creates a workspace. If it did, it would probably be empty.
            Hide
            recampbell Ryan Campbell added a comment -

            This doesn't resolve the issue for me. I see the master computer, but it is not used for the main matrix job. Instead, a cloud slave is provisioned. Steps to repro:

            1. Create a matrix job with a single configuration, no labels
            2. Ensure that Jenkins.numExecutors = 0
            3. Ensure that a cloud plugin can provision slaves
            4. Run the matrix job

            The matrix job won't run on the master, but will result in a slave being provisioned.

            Show
            recampbell Ryan Campbell added a comment - This doesn't resolve the issue for me. I see the master computer, but it is not used for the main matrix job. Instead, a cloud slave is provisioned. Steps to repro: 1. Create a matrix job with a single configuration, no labels 2. Ensure that Jenkins.numExecutors = 0 3. Ensure that a cloud plugin can provision slaves 4. Run the matrix job The matrix job won't run on the master, but will result in a slave being provisioned.
            recampbell Ryan Campbell made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            jglick Jesse Glick added a comment - - edited

            @recampbell: you backported https://github.com/jenkinsci/jenkins/commit/a114c693bc61c564232e78c9fffb5de1ef946ea8, or are testing a 1.510+ build with the original fix, and still see the issue?

            And did you verify that it is the matrix run which waits for, and runs on, the cloud slave instead of master, as opposed to the matrix configuration run which naturally has to run on the cloud slave? (Does testFlyweightTasksWithoutMasterExecutors pass?)

            Show
            jglick Jesse Glick added a comment - - edited @recampbell: you backported https://github.com/jenkinsci/jenkins/commit/a114c693bc61c564232e78c9fffb5de1ef946ea8 , or are testing a 1.510+ build with the original fix, and still see the issue? And did you verify that it is the matrix run which waits for, and runs on, the cloud slave instead of master, as opposed to the matrix configuration run which naturally has to run on the cloud slave? (Does testFlyweightTasksWithoutMasterExecutors pass?)
            Hide
            recampbell Ryan Campbell added a comment -

            Sorry, my mistake in backporting.

            Show
            recampbell Ryan Campbell added a comment - Sorry, my mistake in backporting.
            recampbell Ryan Campbell made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels lts-candidate
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractCIBase.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/model/Queue.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/model/Jenkins.java
            core/src/main/resources/lib/hudson/executors.jelly
            test/src/test/java/hudson/model/QueueTest.java
            http://jenkins-ci.org/commit/jenkins/c5f2d2df58c7781c8003e11b6d389352f4e39adc
            Log:
            [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors.

            Always adding Computer for master as a fallback

            The original proposed fix for JENKINS-7291 creates a Computer object
            transitively. This seems unwise as it violates the design of Computer
            as stated in the javadoc, and for example we can end up creating two
            Computers for the master.

            I think a better fix is to create a Computer for the master all the
            time, even if there's no executors configured. The discrimination in
            Queue.makeBuildable would ensure that such phantom Computer is only used
            as a last resort (statistically speaking).

            I've also tweaked executors.jelly a bit. I simplified it somewhat
            based on the idea that "if there's only one computer to show, the
            context is likely making it obvious".

            (I must be missing the intricacy in the current code.)

            Originally developed in a branch at
            2c5b57fcc1f39ed39057254e802f4183db5aa0dc then squashed for clarity.

            (cherry picked from commit a114c693bc61c564232e78c9fffb5de1ef946ea8)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractCIBase.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/Queue.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/model/Jenkins.java core/src/main/resources/lib/hudson/executors.jelly test/src/test/java/hudson/model/QueueTest.java http://jenkins-ci.org/commit/jenkins/c5f2d2df58c7781c8003e11b6d389352f4e39adc Log: [FIXED JENKINS-7291] Permit flyweight tasks to run on master even when it has zero configured executors. Always adding Computer for master as a fallback The original proposed fix for JENKINS-7291 creates a Computer object transitively. This seems unwise as it violates the design of Computer as stated in the javadoc, and for example we can end up creating two Computers for the master. I think a better fix is to create a Computer for the master all the time, even if there's no executors configured. The discrimination in Queue.makeBuildable would ensure that such phantom Computer is only used as a last resort (statistically speaking). I've also tweaked executors.jelly a bit. I simplified it somewhat based on the idea that "if there's only one computer to show, the context is likely making it obvious". (I must be missing the intricacy in the current code.) Originally developed in a branch at 2c5b57fcc1f39ed39057254e802f4183db5aa0dc then squashed for clarity. (cherry picked from commit a114c693bc61c564232e78c9fffb5de1ef946ea8) Conflicts: changelog.html
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate 1.509.4-fixed
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/model/QueueTest.java
            http://jenkins-ci.org/commit/jenkins/4089c79a68cf56ae57b79431bd795e5d5e5f4e23
            Log:
            JENKINS-7291 Accidental Java 7 dependency in test.
            (cherry picked from commit cedf17e00e1dc60ac4c8f0561e7975b146c9ffd8)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/model/QueueTest.java http://jenkins-ci.org/commit/jenkins/4089c79a68cf56ae57b79431bd795e5d5e5f4e23 Log: JENKINS-7291 Accidental Java 7 dependency in test. (cherry picked from commit cedf17e00e1dc60ac4c8f0561e7975b146c9ffd8)
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 137390 ] JNJira + In-Review [ 187485 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              kohsuke Kohsuke Kawaguchi
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: