Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13619

Add ability to throttle number of concurrent builds of matrix configurations per node

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      It would be extremely useful to be able to throttle the number of concurrent builds of matrix configurations that can take place on a given node. The reason for this is when you have a matrix project in which all subprojects share the same parent workspace data. It is possible then to run concurrent builds across separate nodes, but not on the same node.

      The specific use case I am running into is for building Android. I have a matrix project that defines configurations for each device I wish to build. Every sub-project shares the same android source tree to build off of, so concurrent builds are not possible on the same node as they would clobber each other, but concurrent builds on separate nodes would be fine.

        Attachments

          Issue Links

            Activity

            cvpcs Austen Dicken created issue -
            Hide
            lopou Louis Poudrier-Racette added a comment -

            We are using Jenkins to test code running on embedded devices.
            We have multiple nodes in our setup, each one has only one (1) embedded device attached to it.

            We have two types of Jobs:

            1. Some of our jobs require an embedded devices, they run tests on them. This type of jobs are matrix configurations because there are many versions of the embedded devices to test.
            2. Some of our jobs do not require the embedded devices, they only compile code, they don't run any tests.

            When tests are running on an embedded device (Job type 1), the node's CPU is mostly free. However, we cannot take advantage of this free CPU to run type 2 jobs because we can only have one executor per node.

            We cannot have more then one executor per node because we don't have the "ability to throttle number of concurrent builds of matrix configurations per node" for type 1 jobs.

            We need the "ability to throttle number of concurrent builds of matrix configurations per node" so that different configurations of a type 1 Job are not concurrently run on one node as each node only has one embedded device.

            This would improvement would greatly increase the effectiveness of our continuous integration setup.

            Show
            lopou Louis Poudrier-Racette added a comment - We are using Jenkins to test code running on embedded devices. We have multiple nodes in our setup, each one has only one (1) embedded device attached to it. We have two types of Jobs: Some of our jobs require an embedded devices, they run tests on them. This type of jobs are matrix configurations because there are many versions of the embedded devices to test. Some of our jobs do not require the embedded devices, they only compile code, they don't run any tests. When tests are running on an embedded device (Job type 1), the node's CPU is mostly free. However, we cannot take advantage of this free CPU to run type 2 jobs because we can only have one executor per node. We cannot have more then one executor per node because we don't have the "ability to throttle number of concurrent builds of matrix configurations per node" for type 1 jobs. We need the "ability to throttle number of concurrent builds of matrix configurations per node" so that different configurations of a type 1 Job are not concurrently run on one node as each node only has one embedded device. This would improvement would greatly increase the effectiveness of our continuous integration setup.
            Hide
            wooh Adam PAPAI added a comment - - edited

            Do we have any improvements on this? We also need to throttle our matrix jobs.

            Is there something I can help?

            Show
            wooh Adam PAPAI added a comment - - edited Do we have any improvements on this? We also need to throttle our matrix jobs. Is there something I can help?
            Show
            khneal Kevin Neal added a comment - Would this work? https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin
            Hide
            cjw296 Chris Withers added a comment -

            It appears that the Throttle Concurrent Builds Plugin only works at the parent level, which isn't what is wanted here.

            Show
            cjw296 Chris Withers added a comment - It appears that the Throttle Concurrent Builds Plugin only works at the parent level, which isn't what is wanted here.
            Hide
            cjw296 Chris Withers added a comment -

            I've raised an issue to fix this bug in the Throttle Concurrent Builds Plugin:

            https://issues.jenkins-ci.org/browse/JENKINS-16521

            Show
            cjw296 Chris Withers added a comment - I've raised an issue to fix this bug in the Throttle Concurrent Builds Plugin: https://issues.jenkins-ci.org/browse/JENKINS-16521
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Assignee abayer [ abayer ] Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-16521 [ JENKINS-16521 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java
            src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/config.jelly
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.properties
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c8280ad3ac3c4faf9b69e0401800813a5953dcfa
            Log:
            JENKINS-13619 - Added configuration options for Matrix project's throttling

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.properties http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/c8280ad3ac3c4faf9b69e0401800813a5953dcfa Log: JENKINS-13619 - Added configuration options for Matrix project's throttling Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            dcaroest David Caro added a comment -

            I've tested your branch (https://github.com/jenkinsci/throttle-concurrent-builds-plugin/tree/multiconfig_throttle_support), just had to rebase and remove the ifs that prevent matrix jobs and it works well for me.

            I've also seen this pull request done 5 months ago:
            https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/11

            But haven't tested it yet.

            Is there anything I can do to get it done faster? Should I just send a pull request with the rebased code? Or should I reuse the pull request? If so, what's missing there?

            Thanks!!

            Show
            dcaroest David Caro added a comment - I've tested your branch ( https://github.com/jenkinsci/throttle-concurrent-builds-plugin/tree/multiconfig_throttle_support ), just had to rebase and remove the ifs that prevent matrix jobs and it works well for me. I've also seen this pull request done 5 months ago: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/11 But haven't tested it yet. Is there anything I can do to get it done faster? Should I just send a pull request with the rebased code? Or should I reuse the pull request? If so, what's missing there? Thanks!!
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @David
            Thanks for the reminder.
            I'll merge one of two branches soon

            Show
            oleg_nenashev Oleg Nenashev added a comment - @David Thanks for the reminder. I'll merge one of two branches soon
            oleg_nenashev Oleg Nenashev made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/bdad9a686a8ecf66aad00d17b85388520e935b58
            Log:
            JENKINS-13619 - Display default values in multi-config checkboxes

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/bdad9a686a8ecf66aad00d17b85388520e935b58 Log: JENKINS-13619 - Display default values in multi-config checkboxes Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/739c4a737841c822edae313a1081b6d9cdc91de9
            Log:
            JENKINS-13619 - Check configured options for multi-config throttling

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Conflicts:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/739c4a737841c822edae313a1081b6d9cdc91de9 Log: JENKINS-13619 - Check configured options for multi-config throttling Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com> Conflicts: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/610834a0b68d5ce7489cfaa45193402ea5eb27cb
            Log:
            JENKINS-13619 - Committed missing changes

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Conflicts:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/610834a0b68d5ce7489cfaa45193402ea5eb27cb Log: JENKINS-13619 - Committed missing changes Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com> Conflicts: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/400e7e9f3c67b088e8664d7c89afd7ba70e11dba
            Log:
            JENKINS-13619 - Display default values in multi-config checkboxes

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/400e7e9f3c67b088e8664d7c89afd7ba70e11dba Log: JENKINS-13619 - Display default values in multi-config checkboxes Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/6afbb76840be76864a464fbbb1ecf60ac497e13a
            Log:
            JENKINS-13619 - Check configured options for multi-config throttling

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/6afbb76840be76864a464fbbb1ecf60ac497e13a Log: JENKINS-13619 - Check configured options for multi-config throttling Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/8962bbbf73909f7835af77bde391987c150a2fca
            Log:
            JENKINS-13619 - Committed missing changes

            Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com>

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/c8280ad3ac3c...8962bbbf7390

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/8962bbbf73909f7835af77bde391987c150a2fca Log: JENKINS-13619 - Committed missing changes Signed-off-by: Oleg Nenashev <o.v.nenashev@gmail.com> Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/c8280ad3ac3c...8962bbbf7390
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java
            src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java
            src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/config.jelly
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly
            src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.properties
            src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java
            http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/abd999baf869d0bc598ed6fc522303c629117ed3
            Log:
            Merge pull request #18 from jenkinsci/throttle-matrix-configurations

            JENKINS-13619 - Throttle matrix configurations

            Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/4fa0bb541fbb...abd999baf869

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml src/main/java/hudson/plugins/throttleconcurrents/ThrottleJobProperty.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions.java src/main/java/hudson/plugins/throttleconcurrents/ThrottleQueueTaskDispatcher.java src/main/resources/hudson/plugins/throttleconcurrents/Messages.properties src/main/resources/hudson/plugins/throttleconcurrents/ThrottleJobProperty/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.jelly src/main/resources/hudson/plugins/throttleconcurrents/ThrottleMatrixProjectOptions/config.properties src/test/java/hudson/plugins/throttleconcurrents/ThrottleJobPropertyTest.java http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/abd999baf869d0bc598ed6fc522303c629117ed3 Log: Merge pull request #18 from jenkinsci/throttle-matrix-configurations JENKINS-13619 - Throttle matrix configurations Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/4fa0bb541fbb...abd999baf869
            Show
            oleg_nenashev Oleg Nenashev added a comment - Attached the plugin package from https://jenkins.ci.cloudbees.com/job/plugins/job/throttle-concurrent-builds-plugin/33/org.jenkins-ci.plugins$throttle-concurrents/
            oleg_nenashev Oleg Nenashev made changes -
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-20574 [ JENKINS-20574 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-19986 [ JENKINS-19986 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-23439 [ JENKINS-23439 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Marking as resolved (1.8.3). There will be several improvements in 1.8.4

            Show
            oleg_nenashev Oleg Nenashev added a comment - Marking as resolved (1.8.3). There will be several improvements in 1.8.4
            oleg_nenashev Oleg Nenashev made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-19986 [ JENKINS-19986 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-23902 [ JENKINS-23902 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 144086 ] JNJira + In-Review [ 190903 ]
            Hide
            icando Ke Liu added a comment - - edited

            Sorry to re-surface such an old thread. I am trying to do exactly the same thing. I have a matrix build of N child jobs, and M nodes. Each node can only execute one child job at the same time, while child jobs on different nodes don't interfere with each other. Child jobs from different builds don't interfere with each other as long as they don't run on the same node.

            Is there any example that shows how this can be set up? Right now I observe multiple child jobs being scheduled on the same node. My current configuration (in groovy DSL) is like this (I deleted a bunch of unrelated stuff to make it clearer). Thanks.

             

             

            def numShards = 5
            matrixJob('XXXX') {
              description('XXXX')
              axes {
                text('SHARD', (0..(numShards - 1)).collect { it.toString() })
                label('slaves', "some-label")
              }
              concurrentBuild()
              throttleConcurrentBuilds {
                maxPerNode(1)
              }
              steps {
                shell('some command')
              }
            }
            

             

            Show
            icando Ke Liu added a comment - - edited Sorry to re-surface such an old thread. I am trying to do exactly the same thing. I have a matrix build of N child jobs, and M nodes. Each node can only execute one child job at the same time, while child jobs on different nodes don't interfere with each other. Child jobs from different builds don't interfere with each other as long as they don't run on the same node. Is there any example that shows how this can be set up? Right now I observe multiple child jobs being scheduled on the same node. My current configuration (in groovy DSL) is like this (I deleted a bunch of unrelated stuff to make it clearer). Thanks.     def numShards = 5 matrixJob( 'XXXX' ) {   description( 'XXXX' ) axes {     text( 'SHARD' , (0..(numShards - 1)).collect { it.toString() })    label( 'slaves' , "some-label" )  }  concurrentBuild()  throttleConcurrentBuilds { maxPerNode(1)  }   steps {    shell( 'some command' )   } }  

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              cvpcs Austen Dicken
              Votes:
              7 Vote for this issue
              Watchers:
              22 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: