Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16521

Multi-Project Throttle Categories don't work with Matrix jobs

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I've set up two Multi-Project Throttle Categories, filesystem and postgres, both with maximum total concurrent builds of 0 and maximum concurrent builds per node of 1.

      This is because only one job that uses these categories can run on a host at once, if two try, they'll stomp all over each other.

      So, I've configured two jobs, both matrix jobs, and both configured with "Throttle Concurrent Builds" checked, "throttle this projects as part of one or more catogories" checked and both the categories defined above checked.

      However, when a node has more than one executor, I'll frequently see multiple configurations from the matrix being run on the same node at the same time. Sure enough, they stomp all over each other.

        Attachments

          Issue Links

            Activity

            cjw296 Chris Withers created issue -
            Hide
            cjw296 Chris Withers added a comment -

            This issue would appear to be related:

            https://issues.jenkins-ci.org/browse/JENKINS-12240

            ...but suggests a desire for the opposite behaviour: as many configurations of a particular job as possible should run on a node.

            However, I'm sure there are just as many people who want the behaviour I'm describing, here's an example:

            https://issues.jenkins-ci.org/browse/JENKINS-13619

            Show
            cjw296 Chris Withers added a comment - This issue would appear to be related: https://issues.jenkins-ci.org/browse/JENKINS-12240 ...but suggests a desire for the opposite behaviour: as many configurations of a particular job as possible should run on a node. However, I'm sure there are just as many people who want the behaviour I'm describing, here's an example: https://issues.jenkins-ci.org/browse/JENKINS-13619
            Hide
            cjw296 Chris Withers added a comment -

            I've just verified that this is a regression in 1.7.2.

            1.7.1 behaves as expected, so I'll downgrade to that as a workaround...

            Show
            cjw296 Chris Withers added a comment - I've just verified that this is a regression in 1.7.2. 1.7.1 behaves as expected, so I'll downgrade to that as a workaround...
            Hide
            cjw296 Chris Withers added a comment -
            Show
            cjw296 Chris Withers added a comment - https://issues.jenkins-ci.org/browse/JENKINS-16552 is a duplicate of this issue.
            Hide
            wael Wael Darwich added a comment -

            I tried 1.7.1 but had the same unwanted behaviour as 1.7.2!

            Show
            wael Wael Darwich added a comment - I tried 1.7.1 but had the same unwanted behaviour as 1.7.2!
            Hide
            cjw296 Chris Withers added a comment -

            I'm also now seeing this on 1.7.1 with 1.480.3 :-/

            Show
            cjw296 Chris Withers added a comment - I'm also now seeing this on 1.7.1 with 1.480.3 :-/
            Hide
            judoole Ole Christian Langfjæran added a comment -

            Same here. Version 1.72 and 1.513 of Jenkins. Is this, https://issues.jenkins-ci.org/browse/JENKINS-12240, supposed to fix it? It is included in version 1.72 according to commits.

            Show
            judoole Ole Christian Langfjæran added a comment - Same here. Version 1.72 and 1.513 of Jenkins. Is this, https://issues.jenkins-ci.org/browse/JENKINS-12240 , supposed to fix it? It is included in version 1.72 according to commits.
            Hide
            judoole Ole Christian Langfjæran added a comment -

            Update: Cloned tag 1.7.1 and installed and it seems to work using Category.
            Created a "Matrix-job" category using "Maximum Total Concurrent Build = 4" and "Maximum Concurrent Builds Per Node = 4". The job itself is configured with "Maximum Concurrent Builds Per Node = 1" and "Maximum Total Concurrent Build = 4". I am not using any slaves though.

            Show
            judoole Ole Christian Langfjæran added a comment - Update: Cloned tag 1.7.1 and installed and it seems to work using Category. Created a "Matrix-job" category using "Maximum Total Concurrent Build = 4" and "Maximum Concurrent Builds Per Node = 4". The job itself is configured with "Maximum Concurrent Builds Per Node = 1" and "Maximum Total Concurrent Build = 4". I am not using any slaves though.
            Hide
            cjw296 Chris Withers added a comment -

            Yeah, I thought it was working, and then found overlapping matrix child jobs on some slave nodes.
            So, have to default to one executor per slave, which means those slaves can't be used for any other jobs (that don't use the throttled resources), which is a waste

            Show
            cjw296 Chris Withers added a comment - Yeah, I thought it was working, and then found overlapping matrix child jobs on some slave nodes. So, have to default to one executor per slave, which means those slaves can't be used for any other jobs (that don't use the throttled resources), which is a waste
            Hide
            hiteswar_kumar hiteswar kumar added a comment -

            Hi Andrew(Abayer),
            Can you please fix this and release 1.7.3 fast.
            or if any one has already done fix..please do the pull request to get it merged.

            Show
            hiteswar_kumar hiteswar kumar added a comment - Hi Andrew(Abayer), Can you please fix this and release 1.7.3 fast. or if any one has already done fix..please do the pull request to get it merged.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Reassigned the issue

            Show
            oleg_nenashev Oleg Nenashev added a comment - Reassigned the issue
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Assignee abayer [ abayer ] Oleg Nenashev [ oleg_nenashev ]
            Hide
            swiksted Samu Wikstedt added a comment -

            Hi,
            is this behaviour somehow related to issue we've see lately with matrix jobs and throttle-plugin: throttle plugin do not recognize slave nodes that has been added after the matrix jobs has been gone to queue. Resulting those jobs going to slave nodes that has been there before job launch and new nodes doing nothing.

            Show
            swiksted Samu Wikstedt added a comment - Hi, is this behaviour somehow related to issue we've see lately with matrix jobs and throttle-plugin: throttle plugin do not recognize slave nodes that has been added after the matrix jobs has been gone to queue. Resulting those jobs going to slave nodes that has been there before job launch and new nodes doing nothing.
            Hide
            cjw296 Chris Withers added a comment -

            No, that sounds entirely unrelated. Please file a separate issue for that.

            Show
            cjw296 Chris Withers added a comment - No, that sounds entirely unrelated. Please file a separate issue for that.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I'd say that the issue completely duplicates JENKINS-13619.

            BTW, I've started working on the configurable behavior for Matrix projects. It should resolve both issues.

            Show
            oleg_nenashev Oleg Nenashev added a comment - I'd say that the issue completely duplicates JENKINS-13619 . BTW, I've started working on the configurable behavior for Matrix projects. It should resolve both issues.
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue duplicates JENKINS-13619 [ JENKINS-13619 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            andrea_curtoni Andrea Curtoni added a comment -

            Hi,
            we have another issue with matrix and throttle-concurrents, I cannot say if it is related to this.

            We have 4 executors on a node and 2 matrix projects using them. Each matrix project spawns 4 jobs. Jobs from the same matrix project can run in parallel on the same node but they conflict with jobs from other projects. So we configured the two jobs with the same category and specified to run 1 job per node. Unfortunately jobs from different projects in the same category are spawned to the same node.
            We would expect to have 4 jobs from the same project... or at least 1 job from a single project and 3 free executors, not mixed project jobs.

            We did the following:

            Global configuration:
            Category Name: mock

            • Maximum Total Concurrent Builds: 0
            • Maximum Concurrent Builds Per Node: 1

            Job configurations:
            Throttle Concurrent Builds => checked
            Throttle this project as part of one or more categories => selected

            • Maximum Total Concurrent Builds: 0
            • Maximum Concurrent Builds Per Node: 1
            • Multi-Project Throttle Category: mock

            We also tried to specify the per-node global conf without any help. We also tried to set the limit for the master node (which is used as the main executor for the matrix jobs). No luck.

            Note the same configuration with other non-matrix projects works fine.

            Jenkins version: 1.559
            Throttle Concurrent Builds Plug-in version: 1.8.2

            Should I open another issue for this problem?

            Show
            andrea_curtoni Andrea Curtoni added a comment - Hi, we have another issue with matrix and throttle-concurrents, I cannot say if it is related to this. We have 4 executors on a node and 2 matrix projects using them. Each matrix project spawns 4 jobs. Jobs from the same matrix project can run in parallel on the same node but they conflict with jobs from other projects. So we configured the two jobs with the same category and specified to run 1 job per node. Unfortunately jobs from different projects in the same category are spawned to the same node. We would expect to have 4 jobs from the same project... or at least 1 job from a single project and 3 free executors, not mixed project jobs. We did the following: Global configuration: Category Name: mock Maximum Total Concurrent Builds: 0 Maximum Concurrent Builds Per Node: 1 Job configurations: Throttle Concurrent Builds => checked Throttle this project as part of one or more categories => selected Maximum Total Concurrent Builds: 0 Maximum Concurrent Builds Per Node: 1 Multi-Project Throttle Category: mock We also tried to specify the per-node global conf without any help. We also tried to set the limit for the master node (which is used as the main executor for the matrix jobs). No luck. Note the same configuration with other non-matrix projects works fine. Jenkins version: 1.559 Throttle Concurrent Builds Plug-in version: 1.8.2 Should I open another issue for this problem?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Andrea
            This is a similar issue.
            I'll provide a Dev Build of the plugin in JENKINS-13619

            Show
            oleg_nenashev Oleg Nenashev added a comment - @Andrea This is a similar issue. I'll provide a Dev Build of the plugin in JENKINS-13619
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Marking as duplicate.
            Let's track the further activity in JENKINS-13619

            Show
            oleg_nenashev Oleg Nenashev added a comment - Marking as duplicate. Let's track the further activity in JENKINS-13619
            oleg_nenashev Oleg Nenashev made changes -
            Resolution Duplicate [ 3 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            ircbot Jenkins IRC Bot made changes -
            Component/s matrix-project [ 18765 ]
            Component/s matrix [ 15501 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 147344 ] JNJira + In-Review [ 192349 ]

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              cjw296 Chris Withers
              Votes:
              12 Vote for this issue
              Watchers:
              20 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: