Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58095

Add support for configurable "Restrict jobs execution at node" and "Environment variables" equivalent of generic nodes

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • swarm-plugin
    • None

      I am trying to make our internal build farm more dynamic than a set of predefined SSH agents, and swarm client is quite a successful replacement for us.

      We do however miss some of the generic agents' configurability that we need as different workers are used for different cases.

      One such item is ability to pass Environment variables (and/alternately to Prepare jobs environment, from EnvInject plugin) as part of the swarm client's configuration, perhaps from file handling similar to what is now done for labels. We need it to set up proxies, paths, etc. as applicable to this or that agent deployment. At least, for this half of of the problem we can export the fixed variables from the wrapper script (service method) which starts the swarm client for us, as a workaround. We can not change the definitions without restarting the client, however.

      The other problem is harder to work around: restriction of jobs executions, at least the option to filter by regex of the job name(s). We have use-cases where node labels are good, and some cases where they come short (e.g. we don't want or can't change a node label definition in an experimental project branch, but want that branch practically built on dedicated workers and not intermix with master development - filtering by full-jobname regex supported this use-case well).

          [JENKINS-58095] Add support for configurable "Restrict jobs execution at node" and "Environment variables" equivalent of generic nodes

          Basil Crow added a comment -

          One such item is ability to pass Environment variables (and/alternately to Prepare jobs environment, from EnvInject plugin) as part of the swarm client's configuration

          Can you take a look at jenkinsci/swarm-plugin#80 and see if this doesn't address your use case here?

          The other problem is harder to work around: restriction of jobs executions, at least the option to filter by regex of the job name(s). […] filtering by full-jobname regex supported this use-case well

          When you say "supported" in the past tense, do you mean that you had job name regular expression filtering working already with a static (non-Swarm) agent? I am trying to determine if this problem is unique to Swarm or would apply to any Jenkins agent.

          Basil Crow added a comment - One such item is ability to pass Environment variables (and/alternately to Prepare jobs environment, from EnvInject plugin) as part of the swarm client's configuration Can you take a look at jenkinsci/swarm-plugin#80 and see if this doesn't address your use case here? The other problem is harder to work around: restriction of jobs executions, at least the option to filter by regex of the job name(s). […] filtering by full-jobname regex supported this use-case well When you say "supported" in the past tense, do you mean that you had job name regular expression filtering working already with a static (non-Swarm) agent? I am trying to determine if this problem is unique to Swarm or would apply to any Jenkins agent.

          Michel Zanini added a comment -

          Hi basil,

          Can you take a look at jenkinsci/swarm-plugin#80 and see if this doesn't address your use case here?

          That adds support to Environment variables and works ok, but it has not being officially made part of a release yet. It's not released. It would be useful to release the master branch now to include it.

          When you say "supported" in the past tense, do you mean that you had job name regular expression filtering working already with a static (non-Swarm) agent? I am trying to determine if this problem is unique to Swarm or would apply to any Jenkins agent.

          I believe what he meant here is that he uses https://wiki.jenkins.io/display/JENKINS/Job+Restrictions+Plugin, which implements a regex filter on job names. I also use this plugin and filter. The issue here is that Swarm can't support this as the Node Property gets re-created if the slave disconnects and connects again. It's basically the same issue as with env variables and tool locations, but now in a Node Property coming from a custom plugin.

          I had a look at the code and could not come up with a way we could add support for a Node Property provided by a plugin. The only way I could think would be to dynamically check on server part of the plugin if a certain plugin is installed and add the Node Properties... but the client side has to have a way of sending a big payload configuring the Node Property. For example, Job Restriction plugin contain a list of filters, and the filters need configuration as well such as the regex string. This all has to be sent from the client side and constructed on the server side.

          Seems hard, but do you have any ideas how to support Job Restrictions Plugin with Swarm?

          Michel Zanini added a comment - Hi basil , Can you take a look at jenkinsci/swarm-plugin#80 and see if this doesn't address your use case here? That adds support to Environment variables and works ok, but it has not being officially made part of a release yet. It's not released. It would be useful to release the master branch now to include it. When you say "supported" in the past tense, do you mean that you had job name regular expression filtering working already with a static (non-Swarm) agent? I am trying to determine if this problem is unique to Swarm or would apply to any Jenkins agent. I believe what he meant here is that he uses https://wiki.jenkins.io/display/JENKINS/Job+Restrictions+Plugin , which implements a regex filter on job names. I also use this plugin and filter. The issue here is that Swarm can't support this as the Node Property gets re-created if the slave disconnects and connects again. It's basically the same issue as with env variables and tool locations, but now in a Node Property coming from a custom plugin. I had a look at the code and could not come up with a way we could add support for a Node Property provided by a plugin. The only way I could think would be to dynamically check on server part of the plugin if a certain plugin is installed and add the Node Properties... but the client side has to have a way of sending a big payload configuring the Node Property. For example, Job Restriction plugin contain a list of filters, and the filters need configuration as well such as the regex string. This all has to be sent from the client side and constructed on the server side. Seems hard, but do you have any ideas how to support Job Restrictions Plugin with Swarm?

          Basil Crow added a comment -

          That adds support to Environment variables and works ok, but it has not being officially made part of a release yet. It's not released. It would be useful to release the master branch now to include it.

          I recently released Swarm 3.18, which includes the environment variable support from jenkinsci/swarm-plugin#80.

          Seems hard, but do you have any ideas how to support Job Restrictions Plugin with Swarm?

          It does indeed seem hard, but not impossible. The key is that it would have to be done in a generic way so as to work with all plugins that add node properties. Otherwise, we would just keep having to add special cases for each new plugin that needs to be supported. Does JCasC support the Job Restrictions plugin on an agent definition? If so, then perhaps the Swarm client could be modified to accept some YML input to configure the agent, which could then be sent to the server when creating the agent. The Swarm plugin on the server would then configure the dynamic agent using that YML in the same way that JCasC would configure a static agent. This is all easier said than done; I'm sure that implementing it would be a large effort, but it seems at least theoretically possible.

          Basil Crow added a comment - That adds support to Environment variables and works ok, but it has not being officially made part of a release yet. It's not released. It would be useful to release the master branch now to include it. I recently released Swarm 3.18 , which includes the environment variable support from jenkinsci/swarm-plugin#80 . Seems hard, but do you have any ideas how to support Job Restrictions Plugin with Swarm? It does indeed seem hard, but not impossible. The key is that it would have to be done in a generic way so as to work with all plugins that add node properties. Otherwise, we would just keep having to add special cases for each new plugin that needs to be supported. Does JCasC support the Job Restrictions plugin on an agent definition? If so, then perhaps the Swarm client could be modified to accept some YML input to configure the agent, which could then be sent to the server when creating the agent. The Swarm plugin on the server would then configure the dynamic agent using that YML in the same way that JCasC would configure a static agent. This is all easier said than done; I'm sure that implementing it would be a large effort, but it seems at least theoretically possible.

            Unassigned Unassigned
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: