Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22494

Multiconfiguration project does not respect label restrictions

      When running a multi-configuration project defined in Jenkins, it does not respect the "Label Expression" to "Restrict where this project can be run" under the project's "Advanced Project Options" configuration. If multiple slaves nodes exist, but are not included in the Label Expression for the project, they can still be sent build configurations anyway, which can result in build failures or erroneous builds made in incorrect environments.

      Prerequisites:

      1. Jenkins master installed.
      2. Multiple slave nodes connected to Jenkins master.
      3. Multiple discrete labels to contain nodes.

      Steps to reproduce:

      1. Create new Jenkins project. (Click "New Item".)
      2. Give new item a name, and select Multi-Configuration project by selecting "Build multi-configuration project".
      3. Create at least one user-defined axis with at least two values, e.g. the axis name TestAxis and the values TestValue1 and TestValue2.
      4. Restrict the project to include one node but exclude all other nodes.
        1. Under "Advanced Project Options", tick the checkbox "Restrict where this project can be run"
        2. Refer to the labels you have defined from the above pre-requisite and use a label which refers only to one node. Optionally, use the specific node's name, e.g. if a node is named TestNode1, use this value.
      5. Define a build step of some sort. For simplicity's sake, I used a shell script that simply consisted of one line: echo ${TestAxis}
      6. Run the project.

      Results:
      The project's build configuration will be sent to slave nodes other than the ones explicitly defined in the "Restrict where this project can be run" configuration.

      Expected results:
      Label restrictions are enforced/respected on Multi-Configuration Projects, as they are on other project types.

      Notes:

      • Even using a label expression such as TestNode1 && !TestNode2, project configurations were still sent to TestNode2 to be built.
      • A workaround is to add the Slaves axis to the project's configuration matrix, and select which slaves nodes to use there, and to not use label restriction in Advanced Project Options.
      • This is not a duplicate of the quite similar issue JENKINS-5987, as it described repository checkouts occurring on incorrect nodes, and does not describe the steps listed above.

          [JENKINS-22494] Multiconfiguration project does not respect label restrictions

          Michael Rose added a comment -

          I have noticed the same problem on version 1.560 of Jenkins. I have a multi-configuration project that I restricted to run on nodes with a windows label, but one of the configurations failed b/c it was run on the master (a linux system w/ no labels set).

          Michael Rose added a comment - I have noticed the same problem on version 1.560 of Jenkins. I have a multi-configuration project that I restricted to run on nodes with a windows label, but one of the configurations failed b/c it was run on the master (a linux system w/ no labels set).

          Mark Waite added a comment -

          I believe that the "Restrict where this project can be built" in the Advanced section of a multi-configuration project only limits where the initial flyweight task can execute. I think that the intent is that other areas of the multi-configuration job will be used to control the configuration and the location of the execution tasks. At least, I think that's what I've observed in my use of multi-configuration jobs.

          Mark Waite added a comment - I believe that the "Restrict where this project can be built" in the Advanced section of a multi-configuration project only limits where the initial flyweight task can execute. I think that the intent is that other areas of the multi-configuration job will be used to control the configuration and the location of the execution tasks. At least, I think that's what I've observed in my use of multi-configuration jobs.

          This seems to be borne out with tests of my own; essentially you must use both "Restrict where this project can be built" as Mark Waite says, for the initial task, as well as using the Slaves axis, to control the execution. Forgive me if I have missed it, but, this seems to be documented nowhere.

          Courtland Jones added a comment - This seems to be borne out with tests of my own; essentially you must use both "Restrict where this project can be built" as Mark Waite says, for the initial task, as well as using the Slaves axis, to control the execution. Forgive me if I have missed it, but, this seems to be documented nowhere.

          Mark Waite added a comment - - edited

          My apologies that it isn't documented anywhere. I'm sure I read it somewhere, but I could not find a reference in any of the places I checked. I think that confirms your statement that it is documented nowhere.

          Would you be willing to describe it on the "Building a matrix project wiki page"?

          Mark Waite added a comment - - edited My apologies that it isn't documented anywhere. I'm sure I read it somewhere, but I could not find a reference in any of the places I checked. I think that confirms your statement that it is documented nowhere . Would you be willing to describe it on the " Building a matrix project wiki page "?

          Sure thing, added my edit.
          With this info, I'd consider this issue resolved. Thanks, Mark!

          Courtland Jones added a comment - Sure thing, added my edit . With this info, I'd consider this issue resolved. Thanks, Mark!

          I know this issue is closed, but I would like to add my 2c. Perhaps we should consider re-opening this as a feature request. The jobs of a configuration project should 'inherit' the "Restrict where this project can be built" restrictions. These restrictions should apply in addition to any further restrictions in the matrix axes

          This would have the following advantages:

          1. It's expected behavior. Only restricting the flyweight task surprises most people. (Why would I want to do that anyway?)
          2. Project-wide restrictions ('these jobs need a node with opengl') can be separated from the matrix axes (e.g. release/debug on linux/windows/osx).
          3. The single-value Label Axis work-around would be unnecessary. This work-around is also cumbersome: it makes passing artifacts between projects more 'brittle' since changing a restriction changes the artifact paths for all downstream projects.

          Martin Skinner added a comment - I know this issue is closed, but I would like to add my 2c. Perhaps we should consider re-opening this as a feature request. The jobs of a configuration project should 'inherit' the "Restrict where this project can be built" restrictions. These restrictions should apply in addition to any further restrictions in the matrix axes This would have the following advantages: It's expected behavior. Only restricting the flyweight task surprises most people. (Why would I want to do that anyway?) Project-wide restrictions ('these jobs need a node with opengl') can be separated from the matrix axes (e.g. release/debug on linux/windows/osx). The single-value Label Axis work-around would be unnecessary. This work-around is also cumbersome: it makes passing artifacts between projects more 'brittle' since changing a restriction changes the artifact paths for all downstream projects.

          changing from bug to feature request

          Martin Skinner added a comment - changing from bug to feature request

          Samuel Delisle added a comment - - edited

          Is there any design issue/constraint that prevents a plugin from achieving the proposed behavior? If plugins can control whether or not a node is willing to execute a particular job, it would seem pretty straight-forward to have one that allow you to specify label restrictions (or inherit those of the parent job).

          Samuel Delisle added a comment - - edited Is there any design issue/constraint that prevents a plugin from achieving the proposed behavior? If plugins can control whether or not a node is willing to execute a particular job, it would seem pretty straight-forward to have one that allow you to specify label restrictions (or inherit those of the parent job).

          Mark Waite added a comment -

          mask and samapico I don't see how this could be implemented as an enhancement request without breaking the existing behavior. The existing behavior allows the flyweight task (initial creation of a workspace) to be limited to certain machines, then a completely different (and much richer) set of specifications (extensible by plugins) can be used to control which labels are used to execute the job configurations.

          Today, the first limitation (initial workspace creation) does not apply to the second specification. Considering the number of Jenkins installations, I'm reasonably confident that means there are installations and jobs in those installations which depend on the current behavior.

          Mark Waite added a comment - mask and samapico I don't see how this could be implemented as an enhancement request without breaking the existing behavior. The existing behavior allows the flyweight task (initial creation of a workspace) to be limited to certain machines, then a completely different (and much richer) set of specifications (extensible by plugins) can be used to control which labels are used to execute the job configurations. Today, the first limitation (initial workspace creation) does not apply to the second specification. Considering the number of Jenkins installations, I'm reasonably confident that means there are installations and jobs in those installations which depend on the current behavior.

          I was actually wondering why there did not seem to be any plugin that did this. There are lots of plugin that affect what gets executed where in some way, but I couldn't find any that did this with labels. I finally found this plugin that pretty much does what is said here. You put "return Label1&&Label2" or whatever condition you want in the script box, and it will affect both the parent job and its children.

          Samuel Delisle added a comment - I was actually wondering why there did not seem to be any plugin that did this. There are lots of plugin that affect what gets executed where in some way, but I couldn't find any that did this with labels. I finally found this plugin that pretty much does what is said here. You put "return Label1&&Label2" or whatever condition you want in the script box, and it will affect both the parent job and its children.

          Mark Waite added a comment - - edited

          I use the elastic axis plugin with labels all the time. Coupling that with the platform labeler plugin and the implied labels plugin has allowed me to state that I want a job to run on all Linux and all Windows machines with the expression "linux || windows"

          Mark Waite added a comment - - edited I use the elastic axis plugin with labels all the time. Coupling that with the platform labeler plugin and the implied labels plugin has allowed me to state that I want a job to run on all Linux and all Windows machines with the expression "linux || windows"

          This is basically the same thing I asked for in JENKINS-9531 5 years ago.

          Brian J Murrell added a comment - This is basically the same thing I asked for in JENKINS-9531 5 years ago.

          Another workaround for the issue is to leverage NodeLabel parameter plugin and parameterize the build with node label parameter. That seems to work for us.

          Andrey Myatlyuk added a comment - Another workaround for the issue is to leverage NodeLabel parameter plugin and parameterize the build with node label parameter. That seems to work for us.

          Hermes Cheng added a comment -

          Multiconfiguration project is a default feature of Jenkins v2.0, and I believe this feature is very important for a smooth CI process.

          Hermes Cheng added a comment - Multiconfiguration project is a default feature of Jenkins v2.0, and I believe this feature is very important for a smooth CI process.

          Mark Waite added a comment - - edited

          hermescheng can you please explain why you can't use the elastic axis plugin (as described in my earlier reply) or the NodeLabel parameter plugin to achieve the results you want?

          Also, you may want to refer to my much earlier comment explaining why I believe the multiconfiguration projects behave this way.

          Mark Waite added a comment - - edited hermescheng can you please explain why you can't use the elastic axis plugin (as described in my earlier reply ) or the NodeLabel parameter plugin to achieve the results you want? Also, you may want to refer to my much earlier comment explaining why I believe the multiconfiguration projects behave this way.

          Hermes Cheng added a comment -

          Hi markewaite,
          Yes, elastic axis plugin works well. I just thought as a default feature, it should support the original 'label expression' restricting functionality instead depending on another plugin. Lower the priority since it has a workaround.

          Hermes Cheng added a comment - Hi markewaite , Yes, elastic axis plugin works well. I just thought as a default feature, it should support the original 'label expression' restricting functionality instead depending on another plugin. Lower the priority since it has a workaround.

          Adrian Price added a comment - - edited

          I'm curious as to why this is still open some three years after it was first logged. Isn't it an absolutely fundamental requirement that a build honour its job configuration? As things currently stand the matrix cell jobs get run anywhere, which is surely not the intent behind restricting where the top-level matrix itself can run?

          Adrian Price added a comment - - edited I'm curious as to why this is still open some three years after it was first logged. Isn't it an absolutely fundamental requirement that a build honour its job configuration? As things currently stand the matrix cell jobs get run anywhere, which is surely not the intent behind restricting where the top-level matrix itself can run?

          Mark Waite added a comment -

          adrianp, the multi-configuration project honors the axes assigned to it. If an agent label is used as the condition, that agent label is honored. I don't know why it was reopened, since I believe the code is working as designed, and the design is such that it meets the needs.

          Saying that the "top level matrix runs" the job is a misnomer as far as I can tell, since it is only used for the initial flyweight task, and is not used to perform any of the build actions defined in the job.

          Martin Skinner suggested reopening this as a feature request with the idea that "Restrict where this project can run" should be automatically included in the conditions which define the axes. I disagree with that, since it will be a major compatibility change for existing users, but I suspect that is the reason it has been reopened.

          I consider this to be a "Won't Fix" request, since it is behaving as designed, and the proposed feature request of automatically including the "restrict where this job runs" in the axis definition would break compatibility for some portion of the 80 000+ installations of Jenkins.

          Mark Waite added a comment - adrianp , the multi-configuration project honors the axes assigned to it. If an agent label is used as the condition, that agent label is honored. I don't know why it was reopened, since I believe the code is working as designed, and the design is such that it meets the needs. Saying that the "top level matrix runs" the job is a misnomer as far as I can tell, since it is only used for the initial flyweight task, and is not used to perform any of the build actions defined in the job. Martin Skinner suggested reopening this as a feature request with the idea that "Restrict where this project can run" should be automatically included in the conditions which define the axes. I disagree with that, since it will be a major compatibility change for existing users, but I suspect that is the reason it has been reopened. I consider this to be a "Won't Fix" request, since it is behaving as designed, and the proposed feature request of automatically including the "restrict where this job runs" in the axis definition would break compatibility for some portion of the 80 000+ installations of Jenkins.

          Adrian Price added a comment -

          markewaite, thanks for the quick response. I suppose that in many case one would indeed want the matrix cell jobs to run on different nodes appropriate to particular combinations of matrix axis variables.

          Is there any documentation for your suggestion of using an agent label as the condition? It's not clear to me how to proceed - all I want to do is restrict where the matrix cell jobs run.

          Adrian Price added a comment - markewaite , thanks for the quick response. I suppose that in many case one would indeed want the matrix cell jobs to run on different nodes appropriate to particular combinations of matrix axis variables. Is there any documentation for your suggestion of using an agent label as the condition? It's not clear to me how to proceed - all I want to do is restrict where the matrix cell jobs run.

          Mark Waite added a comment -

          adrianp, if you want to use all agents, or all agents having a particular label, then you can install the elastic axis plugin and configure an axis on the multi-configuration job using elastic axis. Refer to the earlier comment for more on that technique.

          If you want to use specific agents, then define an axis based on the label, and list the labels you want included.

          As an example, for the git plugin, I have a job which defines an "operating-system" axis with values CentOS-6, CenOS-7, Debian-7, Debian-8, Ubuntu-14, Ubuntu-16, windows. Those labels are generated automatically for me by the platformlabeler plugin, so I get good platform coverage without managing label assignments to agents.

          Mark Waite added a comment - adrianp , if you want to use all agents, or all agents having a particular label, then you can install the elastic axis plugin and configure an axis on the multi-configuration job using elastic axis. Refer to the earlier comment for more on that technique. If you want to use specific agents, then define an axis based on the label, and list the labels you want included. As an example, for the git plugin, I have a job which defines an "operating-system" axis with values CentOS-6, CenOS-7, Debian-7, Debian-8, Ubuntu-14, Ubuntu-16, windows. Those labels are generated automatically for me by the platformlabeler plugin, so I get good platform coverage without managing label assignments to agents.

          What if you want to realize something like this:

          Slave axis (os_label): win8, win10

          Slave axis (compiler): msvc10, msvc12

          In our setup it would be many more combinations. How do I make sure, that every combination is only run on slaves that actually have ALL the labels of this combination. E.g. win8 and msvc10.

          In my case it only picks slaves that fulfill the label of the first axis which is IMHO completely unexpected. Am I doing something wrong? Is there a workaround. I feel that this is related to this issue here.

          Julianus Pfeuffer added a comment - What if you want to realize something like this: Slave axis (os_label): win8, win10 Slave axis (compiler): msvc10, msvc12 In our setup it would be many more combinations. How do I make sure, that every combination is only run on slaves that actually have ALL the labels of this combination. E.g. win8 and msvc10. In my case it only picks slaves that fulfill the label of the first axis which is IMHO completely unexpected. Am I doing something wrong? Is there a workaround. I feel that this is related to this issue here.

          Julianus Pfeuffer added a comment - - edited

          I found a workaround on my own by using a combination of the Groovy Label Assignment plugin and the Matrix Groovy Execution Strategy plugin.

          The first uses the script below to make sure all labels to be considered with an "AND".

          if (binding.getVariables().get("os_label") == null) { // for the matrix "parent/flyweight" job
           return "!docker"
          } else {
           return binding.getVariables().get("os_label") + "&&" + binding.getVariables().get("compiler") // for the child jobs
          }

          The second specifies valid combinations by checking available slaves (more complex script). Although it seems to work now, I assumed it to be much easier.

          Julianus Pfeuffer added a comment - - edited I found a workaround on my own by using a combination of the Groovy Label Assignment plugin and the  Matrix Groovy Execution Strategy plugin. The first uses the script below to make sure all labels to be considered with an "AND". if (binding.getVariables().get("os_label") == null) { // for the matrix "parent/flyweight" job  return "!docker" } else {  return binding.getVariables().get("os_label") + "&&" + binding.getVariables().get("compiler") // for the child jobs } The second specifies valid combinations by checking available slaves (more complex script). Although it seems to work now, I assumed it to be much easier.

          I have the following configuration:

          • Upstream matrix job
          • Downstream matrix job (it executed as a post-build action with Parameters set to "Current build parameters")

          These jobs should be executed on separate slave clusters due to project-specific restrictions.

          Jobs contain "Matrix Combinations Parameter", several axes and "Label expression". "Label expression" has the same "Name" for upstream and downstream jobs, but different value for "Label Expressions".

          When upstream job triggered downstream job nothing happend. If I understand correctly, the reason is that Label becomes a part of JOB_NAME variable.

          Is there any way to trigger downstream build on different slave cluster?

          Nikolay Driuchatyi added a comment - I have the following configuration: Upstream matrix job Downstream matrix job (it executed as a post-build action with Parameters set to "Current build parameters") These jobs should be executed on separate slave clusters due to project-specific restrictions. Jobs contain "Matrix Combinations Parameter", several axes and "Label expression". "Label expression" has the same "Name" for upstream and downstream jobs, but different value for "Label Expressions". When upstream job triggered downstream job nothing happend. If I understand correctly, the reason is that Label becomes a part of JOB_NAME variable. Is there any way to trigger downstream build on different slave cluster?

          Arnt Witteveen added a comment - - edited

          I would like to add my reasons on why a way to specify where to run the eventual builds (and where not) is also needed, independent from the axis'.

          I have a master with no compiler, and several slaves with compilers. I want my build to be build in (let's see the example simple) 2 flavors, so I define 1 axis with flavor1 and flavor2 as the value. Every slave can build each of the flavors, they are just different build settings (think building with different defines in C/C++). However, for speed, I would like my 2 builds to run on 2 separate slaves if they are free. And they cannot be run on the master.

          As I see it, there is no solution for this currently: Adding a 'slave' axis makes no sense, I don't want each combination of the other axis to be build on each slave, I only want to build each combination of my axis to be build once. And I don't want to lock certain combinations to certain build machines, these machines might be busy while others are idle, and of course I'd want to use the idle machines then.

           

          My trials may help others:

          • I've tried the 'NodeLabel Parameter Plugin' suggested in JENKINS-23459 . This make the build a paramterized build, which is something I don't want.
          • I had success using the 'Groovy Label Assignment plugin' suggested above, using this script to keep the 'flyweight job' (that starts the axis combination jobs) on the master, and run the actual builds on any node with a label 'CanRunMainBuild' (which could be refined further to run only certain axis on certain node labels, if you need it):

          if (currentJob.getClass().toString() == "class hudson.matrix.MatrixProject")

          { // for the matrix "parent/flyweight" job return "master"; }

          else

          { // for the child jobs return "CanRunMainBuild"; }

          However, now slaves run all the subjobs simultaneously, and the Throttle concurrent builds plugin doesn't block that when using the 'Throttle this project alone' functionality. I think this is JENKINS-33940, and the suggested workaround there (use a category) does work.

          Arnt Witteveen added a comment - - edited I would like to add my reasons on why a way to specify where to run the eventual builds (and where not) is also needed, independent from the axis'. I have a master with no compiler, and several slaves with compilers. I want my build to be build in (let's see the example simple) 2 flavors, so I define 1 axis with flavor1 and flavor2 as the value. Every slave can build each of the flavors, they are just different build settings (think building with different defines in C/C++). However, for speed, I would like my 2 builds to run on 2 separate slaves if they are free. And they cannot be run on the master. As I see it, there is no solution for this currently: Adding a 'slave' axis makes no sense, I don't want each combination of the other axis to be build on each slave, I only want to build each combination of my axis to be build once. And I don't want to lock certain combinations to certain build machines, these machines might be busy while others are idle, and of course I'd want to use the idle machines then.   My trials may help others: I've tried the 'NodeLabel Parameter Plugin' suggested in JENKINS-23459 . This make the build a paramterized build, which is something I don't want. I had success using the 'Groovy Label Assignment plugin' suggested above, using this script to keep the 'flyweight job' (that starts the axis combination jobs) on the master, and run the actual builds on any node with a label 'CanRunMainBuild' (which could be refined further to run only certain axis on certain node labels, if you need it): if (currentJob.getClass().toString() == "class hudson.matrix.MatrixProject") { // for the matrix "parent/flyweight" job return "master"; } else { // for the child jobs return "CanRunMainBuild"; } However, now slaves run all the subjobs simultaneously, and the Throttle concurrent builds plugin doesn't block that when using the 'Throttle this project alone' functionality. I think this is JENKINS-33940 , and the suggested workaround there (use a category) does work.

          Shai Shapira added a comment -

          Any plans to fix this?

          If binding using Node axis, is there any workaround that does not require running all configurations on all nodes with label?

          Shai Shapira added a comment - Any plans to fix this? If binding using Node axis, is there any workaround that does not require running all configurations on all nodes with label?

          Mark Waite added a comment -

          Read the preceding comments for suggestions that allow filtering of matrix configurations

          Mark Waite added a comment - Read the preceding comments for suggestions that allow filtering of matrix configurations

          Shai Shapira added a comment -

          Thanks. I read it, but it does not allow the run of each configuration on one host only.

          i.e. If I have 2 config values, and 4 potential build agents, creating a 2x4 would run 8 jobs, instead of 2.

          Is there a way to solve this?

          Shai Shapira added a comment - Thanks. I read it, but it does not allow the run of each configuration on one host only. i.e. If I have 2 config values, and 4 potential build agents, creating a 2x4 would run 8 jobs, instead of 2. Is there a way to solve this?

          Mark Waite added a comment -

          shai_shap all the details of using exclusions are included in the comments.

          If that's not enough, you might consider switching to the Jenkins pipeline. Blog posts for Declarative pipeline and scripted pipeline describe the techniques to define matrix configurations and to exclude portions of the matrix.

          Mark Waite added a comment - shai_shap all the details of using exclusions are included in the comments. If that's not enough, you might consider switching to the Jenkins pipeline. Blog posts for Declarative pipeline and scripted pipeline describe the techniques to define matrix configurations and to exclude portions of the matrix.

            Unassigned Unassigned
            courtlandj Courtland Jones
            Votes:
            25 Vote for this issue
            Watchers:
            34 Start watching this issue

              Created:
              Updated: