Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22494

Multiconfiguration project does not respect label restrictions

      When running a multi-configuration project defined in Jenkins, it does not respect the "Label Expression" to "Restrict where this project can be run" under the project's "Advanced Project Options" configuration. If multiple slaves nodes exist, but are not included in the Label Expression for the project, they can still be sent build configurations anyway, which can result in build failures or erroneous builds made in incorrect environments.

      Prerequisites:

      1. Jenkins master installed.
      2. Multiple slave nodes connected to Jenkins master.
      3. Multiple discrete labels to contain nodes.

      Steps to reproduce:

      1. Create new Jenkins project. (Click "New Item".)
      2. Give new item a name, and select Multi-Configuration project by selecting "Build multi-configuration project".
      3. Create at least one user-defined axis with at least two values, e.g. the axis name TestAxis and the values TestValue1 and TestValue2.
      4. Restrict the project to include one node but exclude all other nodes.
        1. Under "Advanced Project Options", tick the checkbox "Restrict where this project can be run"
        2. Refer to the labels you have defined from the above pre-requisite and use a label which refers only to one node. Optionally, use the specific node's name, e.g. if a node is named TestNode1, use this value.
      5. Define a build step of some sort. For simplicity's sake, I used a shell script that simply consisted of one line: echo ${TestAxis}
      6. Run the project.

      Results:
      The project's build configuration will be sent to slave nodes other than the ones explicitly defined in the "Restrict where this project can be run" configuration.

      Expected results:
      Label restrictions are enforced/respected on Multi-Configuration Projects, as they are on other project types.

      Notes:

      • Even using a label expression such as TestNode1 && !TestNode2, project configurations were still sent to TestNode2 to be built.
      • A workaround is to add the Slaves axis to the project's configuration matrix, and select which slaves nodes to use there, and to not use label restriction in Advanced Project Options.
      • This is not a duplicate of the quite similar issue JENKINS-5987, as it described repository checkouts occurring on incorrect nodes, and does not describe the steps listed above.

          [JENKINS-22494] Multiconfiguration project does not respect label restrictions

          Adrian Price added a comment -

          markewaite, thanks for the quick response. I suppose that in many case one would indeed want the matrix cell jobs to run on different nodes appropriate to particular combinations of matrix axis variables.

          Is there any documentation for your suggestion of using an agent label as the condition? It's not clear to me how to proceed - all I want to do is restrict where the matrix cell jobs run.

          Adrian Price added a comment - markewaite , thanks for the quick response. I suppose that in many case one would indeed want the matrix cell jobs to run on different nodes appropriate to particular combinations of matrix axis variables. Is there any documentation for your suggestion of using an agent label as the condition? It's not clear to me how to proceed - all I want to do is restrict where the matrix cell jobs run.

          Mark Waite added a comment -

          adrianp, if you want to use all agents, or all agents having a particular label, then you can install the elastic axis plugin and configure an axis on the multi-configuration job using elastic axis. Refer to the earlier comment for more on that technique.

          If you want to use specific agents, then define an axis based on the label, and list the labels you want included.

          As an example, for the git plugin, I have a job which defines an "operating-system" axis with values CentOS-6, CenOS-7, Debian-7, Debian-8, Ubuntu-14, Ubuntu-16, windows. Those labels are generated automatically for me by the platformlabeler plugin, so I get good platform coverage without managing label assignments to agents.

          Mark Waite added a comment - adrianp , if you want to use all agents, or all agents having a particular label, then you can install the elastic axis plugin and configure an axis on the multi-configuration job using elastic axis. Refer to the earlier comment for more on that technique. If you want to use specific agents, then define an axis based on the label, and list the labels you want included. As an example, for the git plugin, I have a job which defines an "operating-system" axis with values CentOS-6, CenOS-7, Debian-7, Debian-8, Ubuntu-14, Ubuntu-16, windows. Those labels are generated automatically for me by the platformlabeler plugin, so I get good platform coverage without managing label assignments to agents.

          What if you want to realize something like this:

          Slave axis (os_label): win8, win10

          Slave axis (compiler): msvc10, msvc12

          In our setup it would be many more combinations. How do I make sure, that every combination is only run on slaves that actually have ALL the labels of this combination. E.g. win8 and msvc10.

          In my case it only picks slaves that fulfill the label of the first axis which is IMHO completely unexpected. Am I doing something wrong? Is there a workaround. I feel that this is related to this issue here.

          Julianus Pfeuffer added a comment - What if you want to realize something like this: Slave axis (os_label): win8, win10 Slave axis (compiler): msvc10, msvc12 In our setup it would be many more combinations. How do I make sure, that every combination is only run on slaves that actually have ALL the labels of this combination. E.g. win8 and msvc10. In my case it only picks slaves that fulfill the label of the first axis which is IMHO completely unexpected. Am I doing something wrong? Is there a workaround. I feel that this is related to this issue here.

          Julianus Pfeuffer added a comment - - edited

          I found a workaround on my own by using a combination of the Groovy Label Assignment plugin and the Matrix Groovy Execution Strategy plugin.

          The first uses the script below to make sure all labels to be considered with an "AND".

          if (binding.getVariables().get("os_label") == null) { // for the matrix "parent/flyweight" job
           return "!docker"
          } else {
           return binding.getVariables().get("os_label") + "&&" + binding.getVariables().get("compiler") // for the child jobs
          }

          The second specifies valid combinations by checking available slaves (more complex script). Although it seems to work now, I assumed it to be much easier.

          Julianus Pfeuffer added a comment - - edited I found a workaround on my own by using a combination of the Groovy Label Assignment plugin and the  Matrix Groovy Execution Strategy plugin. The first uses the script below to make sure all labels to be considered with an "AND". if (binding.getVariables().get("os_label") == null) { // for the matrix "parent/flyweight" job  return "!docker" } else {  return binding.getVariables().get("os_label") + "&&" + binding.getVariables().get("compiler") // for the child jobs } The second specifies valid combinations by checking available slaves (more complex script). Although it seems to work now, I assumed it to be much easier.

          I have the following configuration:

          • Upstream matrix job
          • Downstream matrix job (it executed as a post-build action with Parameters set to "Current build parameters")

          These jobs should be executed on separate slave clusters due to project-specific restrictions.

          Jobs contain "Matrix Combinations Parameter", several axes and "Label expression". "Label expression" has the same "Name" for upstream and downstream jobs, but different value for "Label Expressions".

          When upstream job triggered downstream job nothing happend. If I understand correctly, the reason is that Label becomes a part of JOB_NAME variable.

          Is there any way to trigger downstream build on different slave cluster?

          Nikolay Driuchatyi added a comment - I have the following configuration: Upstream matrix job Downstream matrix job (it executed as a post-build action with Parameters set to "Current build parameters") These jobs should be executed on separate slave clusters due to project-specific restrictions. Jobs contain "Matrix Combinations Parameter", several axes and "Label expression". "Label expression" has the same "Name" for upstream and downstream jobs, but different value for "Label Expressions". When upstream job triggered downstream job nothing happend. If I understand correctly, the reason is that Label becomes a part of JOB_NAME variable. Is there any way to trigger downstream build on different slave cluster?

          Arnt Witteveen added a comment - - edited

          I would like to add my reasons on why a way to specify where to run the eventual builds (and where not) is also needed, independent from the axis'.

          I have a master with no compiler, and several slaves with compilers. I want my build to be build in (let's see the example simple) 2 flavors, so I define 1 axis with flavor1 and flavor2 as the value. Every slave can build each of the flavors, they are just different build settings (think building with different defines in C/C++). However, for speed, I would like my 2 builds to run on 2 separate slaves if they are free. And they cannot be run on the master.

          As I see it, there is no solution for this currently: Adding a 'slave' axis makes no sense, I don't want each combination of the other axis to be build on each slave, I only want to build each combination of my axis to be build once. And I don't want to lock certain combinations to certain build machines, these machines might be busy while others are idle, and of course I'd want to use the idle machines then.

           

          My trials may help others:

          • I've tried the 'NodeLabel Parameter Plugin' suggested in JENKINS-23459 . This make the build a paramterized build, which is something I don't want.
          • I had success using the 'Groovy Label Assignment plugin' suggested above, using this script to keep the 'flyweight job' (that starts the axis combination jobs) on the master, and run the actual builds on any node with a label 'CanRunMainBuild' (which could be refined further to run only certain axis on certain node labels, if you need it):

          if (currentJob.getClass().toString() == "class hudson.matrix.MatrixProject")

          { // for the matrix "parent/flyweight" job return "master"; }

          else

          { // for the child jobs return "CanRunMainBuild"; }

          However, now slaves run all the subjobs simultaneously, and the Throttle concurrent builds plugin doesn't block that when using the 'Throttle this project alone' functionality. I think this is JENKINS-33940, and the suggested workaround there (use a category) does work.

          Arnt Witteveen added a comment - - edited I would like to add my reasons on why a way to specify where to run the eventual builds (and where not) is also needed, independent from the axis'. I have a master with no compiler, and several slaves with compilers. I want my build to be build in (let's see the example simple) 2 flavors, so I define 1 axis with flavor1 and flavor2 as the value. Every slave can build each of the flavors, they are just different build settings (think building with different defines in C/C++). However, for speed, I would like my 2 builds to run on 2 separate slaves if they are free. And they cannot be run on the master. As I see it, there is no solution for this currently: Adding a 'slave' axis makes no sense, I don't want each combination of the other axis to be build on each slave, I only want to build each combination of my axis to be build once. And I don't want to lock certain combinations to certain build machines, these machines might be busy while others are idle, and of course I'd want to use the idle machines then.   My trials may help others: I've tried the 'NodeLabel Parameter Plugin' suggested in JENKINS-23459 . This make the build a paramterized build, which is something I don't want. I had success using the 'Groovy Label Assignment plugin' suggested above, using this script to keep the 'flyweight job' (that starts the axis combination jobs) on the master, and run the actual builds on any node with a label 'CanRunMainBuild' (which could be refined further to run only certain axis on certain node labels, if you need it): if (currentJob.getClass().toString() == "class hudson.matrix.MatrixProject") { // for the matrix "parent/flyweight" job return "master"; } else { // for the child jobs return "CanRunMainBuild"; } However, now slaves run all the subjobs simultaneously, and the Throttle concurrent builds plugin doesn't block that when using the 'Throttle this project alone' functionality. I think this is JENKINS-33940 , and the suggested workaround there (use a category) does work.

          Shai Shapira added a comment -

          Any plans to fix this?

          If binding using Node axis, is there any workaround that does not require running all configurations on all nodes with label?

          Shai Shapira added a comment - Any plans to fix this? If binding using Node axis, is there any workaround that does not require running all configurations on all nodes with label?

          Mark Waite added a comment -

          Read the preceding comments for suggestions that allow filtering of matrix configurations

          Mark Waite added a comment - Read the preceding comments for suggestions that allow filtering of matrix configurations

          Shai Shapira added a comment -

          Thanks. I read it, but it does not allow the run of each configuration on one host only.

          i.e. If I have 2 config values, and 4 potential build agents, creating a 2x4 would run 8 jobs, instead of 2.

          Is there a way to solve this?

          Shai Shapira added a comment - Thanks. I read it, but it does not allow the run of each configuration on one host only. i.e. If I have 2 config values, and 4 potential build agents, creating a 2x4 would run 8 jobs, instead of 2. Is there a way to solve this?

          Mark Waite added a comment -

          shai_shap all the details of using exclusions are included in the comments.

          If that's not enough, you might consider switching to the Jenkins pipeline. Blog posts for Declarative pipeline and scripted pipeline describe the techniques to define matrix configurations and to exclude portions of the matrix.

          Mark Waite added a comment - shai_shap all the details of using exclusions are included in the comments. If that's not enough, you might consider switching to the Jenkins pipeline. Blog posts for Declarative pipeline and scripted pipeline describe the techniques to define matrix configurations and to exclude portions of the matrix.

            Unassigned Unassigned
            courtlandj Courtland Jones
            Votes:
            25 Vote for this issue
            Watchers:
            34 Start watching this issue

              Created:
              Updated: