-
New Feature
-
Resolution: Won't Do
-
Minor
-
None
-
Build enviornment:
Jenkins ver. 1.625.3.2 (CloudBees Jenkins Enterprise 15.11)
docker-plugin 0.16.0
When building a Matrix project we would like to be able to have the build spin up docker containers for each flyweight task that the matrix project creates. I would like to set master executors to 0 and only allow building inside of docker containers. So each task would create a container and the execute inside of that container.
I would like to remain more flexible with matrix builds and be able to adjust my build environment to suit what the build is actually doing.
- links to
[JENKINS-34238] Flyweight Job Support For Docker Plugin
To clarify, the flyweight task is the task coordinates the sub-tasks and assembles the final results. Each of the sub-tasks already run properly in docker containers. But the coordinating flyweight task does not run in a docker container (the ability is explicitly disabled with a conditional last I checked the docker plugin source).
Therefore, if the coordinating flyweight task doesn't have a place (e.g. another container or permanent host) to run it then none of the sub-tasks are created.
node() {} blocks aren't very good at matrix building without a lot of complicated scripting. (i.e. saving the script as a variable and reusing it half a dozen times in node blocks). pipeline also doesn't cleanly assemble the results like the matrix building project type can. pipeline plugin is not a good solution for this IMO or current functionality is lacking.
Sam Gleske
added a comment - - edited To clarify, the flyweight task is the task coordinates the sub-tasks and assembles the final results. Each of the sub-tasks already run properly in docker containers. But the coordinating flyweight task does not run in a docker container (the ability is explicitly disabled with a conditional last I checked the docker plugin source).
Therefore, if the coordinating flyweight task doesn't have a place (e.g. another container or permanent host) to run it then none of the sub-tasks are created.
node() {} blocks aren't very good at matrix building without a lot of complicated scripting. (i.e. saving the script as a variable and reusing it half a dozen times in node blocks). pipeline also doesn't cleanly assemble the results like the matrix building project type can. pipeline plugin is not a good solution for this IMO or current functionality is lacking. | Component/s | New: matrix-project-plugin [ 18765 ] |
| Workflow | Original: JNJira [ 170298 ] | New: JNJira + In-Review [ 183846 ] |
| Assignee | Original: magnayn [ magnayn ] | New: Nicolas De Loof [ ndeloof ] |
| Remote Link | New: This issue links to "CloudBees Internal OSS-1009 (Web Link)" [ 18788 ] |
pjdarton
added a comment - I have "# of executors" set to zero for my master and it runs flyweight tasks perfectly ok.
i.e. you *can *set master executors to 0 already.
Flyweight tasks do not take up an executor, so "zero" is a perfectly acceptable number of executors to permit flyweight tasks.
Running flyweight tasks on dynamically-provisioned slaves isn't a good idea as they're likely to get disposed of the moment that the non-flyweight task (that they were created for) completes.
e.g. the vSphere-plugin doesn't allow flyweight tasks on its slaves because they can get closed down while the flyweight task is still running.
pjdarton
added a comment - I have "# of executors" set to zero for my master and it runs flyweight tasks perfectly ok.
i.e. you *can *set master executors to 0 already.
Flyweight tasks do not take up an executor, so "zero" is a perfectly acceptable number of executors to permit flyweight tasks.
Running flyweight tasks on dynamically-provisioned slaves isn't a good idea as they're likely to get disposed of the moment that the non-flyweight task (that they were created for) completes.
e.g. the vSphere-plugin doesn't allow flyweight tasks on its slaves because they can get closed down while the flyweight task is still running. This feature no longer affects me in project matrix job types. I've since moved on to the pipeline plugin in my open source projects which does matrix building just fine. A lot has changed since I made that comment in 2016. Pipelines have improved. Blue ocean was released so parallel matrix building is no longer ugly.
You can see how I set up matrix jobs in pipeline similar to how they used to be done in my matrix project type jobs.
Sam Gleske
added a comment - - edited This feature no longer affects me in project matrix job types. I've since moved on to the pipeline plugin in my open source projects which does matrix building just fine. A lot has changed since I made that comment in 2016. Pipelines have improved. Blue ocean was released so parallel matrix building is no longer ugly.
You can see how I set up matrix jobs in pipeline similar to how they used to be done in my matrix project type jobs.
https://github.com/samrocketman/jervis/blob/f283197fa9267429b5bca84c376e43edb46e6d03/vars/buildViaJervis.groovy#L223-L268 pjdarton
added a comment - Closing as it's not required anymore. pjdarton
made changes -
| Assignee | Original: Nicolas De Loof [ ndeloof ] | |
| Resolution | New: Won't Do [ 10001 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |
Sounds to me like you want to be looking at pipelines and sections within node() {} blocks.
docker plugin merely implements the cloud API, so there is no forseeable chance if implementing what you're asking for without a significant core change.