Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48050

Replace Declarative Docker agent directive with new implementation

    • Declarative - 1.2, Pipeline - December, Pipeline - April 2018

      Goals

      • We want to have a new implementation for running steps inside a Docker container that is transparent to the Pipeline author
      • We want to ensure that as this developed that we start making changes that allow us to swap out the implementation in the future (e.g. docker, k8, etc)
      • We want customers to try this new implementation without having to use patched plugins or special builds from a branch

      Scope

      • Declarative Docker agent directive can be used to run steps for a stage or for the whole pipeline
      • The new implementation can be enabled by specifying Pipeline->Options->dockerPipelineVersion = 2
      • Must be able to use an off the shelf docker container without modifying it so it works with Jenkins
      • Honour the entry point of the container
      • Sharing the workspace from the top level agent to a per stage Docker agent (and back again!)

      Out of scope

      • Declarative Docker agent directive that specifies a Dockerfile
      • Scripted docker.inside
      • Scripted docker.build
      • Building containers
      • Linked containers
      • Caching directories

      Acceptance criteria
      The following steps should work using the new implementation (even if modifications are needed to these steps):

      • withMaven()
      • dir()
      • withCredentials

      The following functionality should work:

      • Tool installers
      • Config File Provider

      Examples
      Per stage

      pipeline {
        options {
          dockerPipelineVersion(2)
        }
        agent none // Will not reuse workspace
        stages {
           stage (‘build’) {
              agent { docker ‘maven3’ } 
              steps { 
      	   sh ‘mvn clean test’
              } 
           }
        }
      }
      

      Whole Pipeline

      pipeline {
        options {
          dockerPipelineVersion(2)
        }
        agent { docker ‘maven3’ } // will reuse workspace for all stages unless specified
        stages {
           stage (‘build’) {
              steps { 
      	   sh ‘mvn clean test’
              } 
           }
        }
      }
      

          [JENKINS-48050] Replace Declarative Docker agent directive with new implementation

          Andrew Bayer added a comment -

          A couple more criteria I’d like to suggest:

          • Doesn’t require any additional configuration on the master. This is a key aspect of the existing behavior, so I’d say it really needs to be retained.
          • JENKINS-46831 is fixed - that is, it’s possible to have one docker agent at the top level and a different one on a stage. This’ll probably come for free with dir() working and the like, but it’s still worth mentioning.

          The scope here also doesn’t address the question of sharing the workspace from the top level agent to a per stage Docker agent (and back again!). I think that’s important and should be retained, but not as important as the lack of additional master configuration.

          Andrew Bayer added a comment - A couple more criteria I’d like to suggest: Doesn’t require any additional configuration on the master. This is a key aspect of the existing behavior, so I’d say it really needs to be retained. JENKINS-46831 is fixed - that is, it’s possible to have one docker agent at the top level and a different one on a stage. This’ll probably come for free with dir() working and the like, but it’s still worth mentioning. The scope here also doesn’t address the question of sharing the workspace from the top level agent to a per stage Docker agent (and back again!). I think that’s important and should be retained, but not as important as the lack of additional master configuration.

          James Dumay added a comment - - edited

          abayer

          Doesn’t require any additional configuration on the master

          I suppose this means everything falls back to using the docker socket if Jenkins doesn't provide an implementation?

          The scope here also doesn’t address the question of sharing the workspace from the top level agent to a per stage Docker agent

          Yes, it should. Ill add this to the description.

          James Dumay added a comment - - edited abayer Doesn’t require any additional configuration on the master I suppose this means everything falls back to using the docker socket if Jenkins doesn't provide an implementation? The scope here also doesn’t address the question of sharing the workspace from the top level agent to a per stage Docker agent Yes, it should. Ill add this to the description.

          James Nord added a comment -

          I would suggest adding withCredentials to the scope as without credentials many things are dead in the water.

          I would also suggest that any new implementation should be /aware/ of windows containers (ie. .its fine if it doesn't work with them on day zero, but if it needs architectural changes to work with them in the future it should be a showstopper. (this may involve things like flagging the desired agent type so that you can choose the correct agent on which to launch the container).

          Doesn’t require any additional configuration on the master. This is a key aspect of the existing behavior, so I’d say it really needs to be retained.

          well today you 100% have to do this, because not all agents (even if they are cattle) are the same and some herds will not be docker enabled and if you do not also specify a label then you will eventually go bang, when the farmer introduces a new breed of cattle (or perhaps branches out into pig farming)

          James Nord added a comment - I would suggest adding withCredentials to the scope as without credentials many things are dead in the water. I would also suggest that any new implementation should be /aware/ of windows containers (ie. .its fine if it doesn't work with them on day zero, but if it needs architectural changes to work with them in the future it should be a showstopper. (this may involve things like flagging the desired agent type so that you can choose the correct agent on which to launch the container). Doesn’t require any additional configuration on the master. This is a key aspect of the existing behavior, so I’d say it really needs to be retained. well today you 100% have to do this, because not all agents (even if they are cattle) are the same and some herds will not be docker enabled and if you do not also specify a label then you will eventually go bang, when the farmer introduces a new breed of cattle (or perhaps branches out into pig farming)

          Cyrille Le Clerc added a comment - - edited

          Can we add some criterion of feature parity of dockerized build agents with "permanent jenkins agents":

          • Tool installers
          • Environment variable contribution by plugins, including the PATH variable
          • Config file provider plugin
          • Caching of some directories: maven/gradle repo, npm... in a mode that can be extended for other programming languages

          Note that some of these criterion are implied by the criterion to support withMaven

          Cyrille Le Clerc added a comment - - edited Can we add some criterion of feature parity of dockerized build agents with "permanent jenkins agents": Tool installers Environment variable contribution by plugins, including the PATH variable Config file provider plugin Caching of some directories: maven/gradle repo, npm... in a mode that can be extended for other programming languages Note that some of these criterion are implied by the criterion to support withMaven

          Andrew Bayer added a comment -

          withCredentials, tool installers, and variables on the agent like PATH should definitely work - I think in practice, all of those will work without any additional effort so long as we're interacting with the container as a true agent behind the scenes, as discussed already, but it'll definitely be worth having tests for those.

          re: directory caching - I'd say that comes later.

          Andrew Bayer added a comment - withCredentials , tool installers, and variables on the agent like PATH should definitely work - I think in practice, all of those will work without any additional effort so long as we're interacting with the container as a true agent behind the scenes, as discussed already, but it'll definitely be worth having tests for those. re: directory caching - I'd say that comes later.

          Cyrille Le Clerc added a comment - - edited

          jamesdumay as discussed with abayer, I feel that we should restrict the list of supported credentials types for withCredentials because it's the responsibility of each credentials plugin to implement the "not very well documented" contract that files created by plugins for builds should be created in hudson.slaves.WorkspaceList#tempDir. I propose to restrict to

          • secret file: should be ok because it's implemented by the credentials binding plugin directly
          • secret zip file: should be ok because it's implemented by the credentials binding plugin directly
          • docker client certificate: provided by the docker commons plugin
          • AWS access key: provided by the AWS Credentials Plugin, nothing to test it's environment variables, not files
          • username and password conjoined and separated: provided by the Credentials Binding Plugin, nothing to test it's environment variables, not files
          • SSH private key: I didn't find how it's handled by withCredentials
          • GPG certs: I didn't find how it's handled by withCredentials

          Moreover, can we clarify/document the behavior of the declarative docker agent regarding:

          • The "tmp" dir of the JVM of the agent. docker-pipeline decided to not mount it, do we confirm this choice? Not mounting the temp folder of the build agent JVM has the benefit of providing more isolation of the builds but has the drawback of breaking a bunch of plugin (was breaking the config file provider plugin and some credentials files)
          • The mount / or not mount of the tools folder. docker-pipeline-plugin decided to not mounting it (breaking tool installers), custom-build-environment-plugin decided to mount this folder, preserving feature parity on tool installers.

          Last but not least, we could check that the following things works

          • "git" and "checkout" using an ssh private key
          • "sshAgent": I don't know if it relies on a file on disk

          Docker things

          • Can we confirm that we fix JENKINS-41316 "docker.image('my-image').inside{...} no longer honors Dockerfile "entrypoint" since version 1.8"

          Pending questions:

          • OpenShift credentials by the kubernetes plugin: see OpenShiftTokenCredentialImpl.java,
            • Doesn't seem to be integrated with withCredentials
          • Kubernetes plugin seem to rely on disk on
          • perforce credentials by the P4 plugin

          Cyrille Le Clerc added a comment - - edited jamesdumay as discussed with abayer , I feel that we should restrict the list of supported credentials types for withCredentials because it's the responsibility of each credentials plugin to implement the "not very well documented" contract that files created by plugins for builds should be created in hudson.slaves.WorkspaceList#tempDir . I propose to restrict to secret file: should be ok because it's implemented by the credentials binding plugin directly secret zip file: should be ok because it's implemented by the credentials binding plugin directly docker client certificate: provided by the docker commons plugin AWS access key: provided by the AWS Credentials Plugin, nothing to test it's environment variables, not files username and password conjoined and separated: provided by the Credentials Binding Plugin, nothing to test it's environment variables, not files SSH private key: I didn't find how it's handled by withCredentials GPG certs: I didn't find how it's handled by withCredentials Moreover, can we clarify/document the behavior of the declarative docker agent regarding: The "tmp" dir of the JVM of the agent. docker-pipeline decided to not mount it, do we confirm this choice? Not mounting the temp folder of the build agent JVM has the benefit of providing more isolation of the builds but has the drawback of breaking a bunch of plugin (was breaking the config file provider plugin and some credentials files) The mount / or not mount of the tools folder. docker-pipeline-plugin decided to not mounting it (breaking tool installers), custom-build-environment-plugin decided to mount this folder, preserving feature parity on tool installers. Last but not least, we could check that the following things works " git " and " checkout " using an ssh private key "sshAgent": I don't know if it relies on a file on disk Docker things Can we confirm that we fix JENKINS-41316 "docker.image('my-image').inside{...} no longer honors Dockerfile "entrypoint" since version 1.8" Pending questions: OpenShift credentials by the kubernetes plugin: see OpenShiftTokenCredentialImpl.java , Doesn't seem to be integrated with withCredentials Kubernetes plugin seem to rely on disk on See ServiceAccountCredential.java and SERVICEACCOUNT_TOKEN_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/token" perforce credentials by the P4 plugin

          Jesse Glick added a comment -

          Huh? Any kind of credentials with a Binding implementation is supported by withCredentials.

          Jesse Glick added a comment - Huh? Any kind of credentials with a Binding implementation is supported by withCredentials .

          Jesse Glick added a comment -

          And of course docker-workflow mounts the workspace temp dir. Tons of stuff would break otherwise.

          Mounting a tools directory is unsafe because it is shared across builds.

          I think you need me on design review.

          Jesse Glick added a comment - And of course docker-workflow mounts the workspace temp dir. Tons of stuff would break otherwise. Mounting a tools directory is unsafe because it is shared across builds. I think you need me on design review.

          Jesse Glick added a comment -

          checkout with GitSCM ignores any decorated Launcher due to a well-known design flaw in the git-client plugin, so this is irrelevant. Test with a properly implemented SCM, like mercurial.

          Jesse Glick added a comment - checkout with GitSCM ignores any decorated Launcher due to a well-known design flaw in the git-client plugin, so this is irrelevant. Test with a properly implemented SCM, like mercurial .

          Andrew Bayer added a comment -

          The plan is that there will be a true Docker agent - i.e., it'll mount in the workspace but nothing else from the underlying host, so anything that works on, say, a container agent (via the k8s plugin, or Docker plugin's cloud, or Azure Container Service, or...you get the point), will work the same on this new agent as well.

          So let's not worry about complexity of edge cases that don't work in docker-workflow due to it not being an actual agent - that's why dir, tools, etc don't work, because they expect to be acting on the agent, not inside the container within the agent. When we're interacting with the container as a true agent, running remoting.jar and all, those problems should just vanish. And to be sure, we'll be adding tests for a wide array of behaviors, both ones that don't work on docker-workflow and ones that do, of course.

          Andrew Bayer added a comment - The plan is that there will be a true Docker agent - i.e., it'll mount in the workspace but nothing else from the underlying host, so anything that works on, say, a container agent (via the k8s plugin, or Docker plugin's cloud, or Azure Container Service, or...you get the point), will work the same on this new agent as well. So let's not worry about complexity of edge cases that don't work in docker-workflow due to it not being an actual agent - that's why dir , tools, etc don't work, because they expect to be acting on the agent, not inside the container within the agent. When we're interacting with the container as a true agent, running remoting.jar and all, those problems should just vanish. And to be sure, we'll be adding tests for a wide array of behaviors, both ones that don't work on docker-workflow and ones that do, of course.

          James Dumay added a comment - - edited

          ndeloof you mentioned some changes in a recent docker release that could be beneficial to this work. Would you be able to find out roughly when this change gets into a Docker Enterprise release?

          Random thought: if Docker is too old, then use the current Docker Pipeline implementation or fit its new enough use the new Docker implementation?

          James Dumay added a comment - - edited ndeloof you mentioned some changes in a recent docker release that could be beneficial to this work. Would you be able to find out roughly when this change gets into a Docker Enterprise release? Random thought: if Docker is too old, then use the current Docker Pipeline implementation or fit its new enough use the new Docker implementation?

          rsandell added a comment -

          So why should we do this?

          I see goals and acceptance criteria, but no motivation on why this is needed?

          rsandell added a comment - So why should we do this? I see goals and acceptance criteria, but no motivation on why this is needed?

          James Dumay added a comment -

          The current implementation is broken by design. Steps like dir and withMaven do not work

          James Dumay added a comment - The current implementation is broken by design. Steps like dir and withMaven do not work

          James Dumay added a comment -

          Also tool installers

          James Dumay added a comment - Also tool installers

          Andrew Bayer added a comment -

          fwiw, here are some JIRAs that are basically unfixable due to the architecture of Docker Pipeline (a few of them may be fixed by https://github.com/jenkinsci/docker-workflow-plugin/pull/123, but that requires Docker 17.12 and none of them would be an issue if we were actually running the agent process on the container itself anyway):

          Then there's the issue of Docker Pipeline's maintainability - there are a number of things about its design and implementation that are now considered mistakes, it mashes up a lot of different functionality into one codebase that can be hard to unravel, the docker.image(...).inside syntax is problematic, etc...

          Andrew Bayer added a comment - fwiw, here are some JIRAs that are basically unfixable due to the architecture of Docker Pipeline (a few of them may be fixed by https://github.com/jenkinsci/docker-workflow-plugin/pull/123 , but that requires Docker 17.12 and none of them would be an issue if we were actually running the agent process on the container itself anyway): JENKINS-46831 JENKINS-33510 JENKINS-41894 JENKINS-47890 JENKINS-47805 JENKINS-43590 JENKINS-41316 JENKINS-36913 JENKINS-48082 JENKINS-48319 Then there's the issue of Docker Pipeline's maintainability - there are a number of things about its design and implementation that are now considered mistakes, it mashes up a lot of different functionality into one codebase that can be hard to unravel, the docker.image(...).inside syntax is problematic, etc...

          Steve Todorov added a comment -

          This seems like it will fix a lot of the issues we're having with running maven builds in docker containers.
          What is the approximate timeline for having this feature released - I'm guessing a few months? 

          Steve Todorov added a comment - This seems like it will fix a lot of the issues we're having with running maven builds in docker containers. What is the approximate timeline for having this feature released - I'm guessing a few months? 

          Nothing planned yet, not even agreement on implementation decisions. 

          Nicolas De Loof added a comment - Nothing planned yet, not even agreement on implementation decisions. 

          Steve Todorov added a comment -

          Thanks for your reply. Please keep us in the loop in that case.  

          Steve Todorov added a comment - Thanks for your reply. Please keep us in the loop in that case.  

          Waldek M added a comment -

          Hello,

          I'm wondering  - would the new implementation be also solving problems like https://issues.jenkins-ci.org/browse/JENKINS-47026 ?

          I.e. would it also honor the isolation of Docker container from host (processes running within and outside of Docker container with different UID/GID)?

          Waldek M added a comment - Hello, I'm wondering  - would the new implementation be also solving problems like https://issues.jenkins-ci.org/browse/JENKINS-47026 ? I.e. would it also honor the isolation of Docker container from host (processes running within and outside of Docker container with different UID/GID)?

          weakcamel this is uncertain so far, as this issue is more about balancing what we'd like to get and what is feasible . Typically

          • we'd like to be able to run arbitrary docker image but jenkins require a JVM for it's agent
          • we'd like to be able to share workspace between steps running in containers, but then the uid/gid issue will come back 
          • etc ... always with but xx at some point

          IMHO we need to define a fresh new pipeline approach for docker usage that would not try to hack docker container. Working on a proposal on this purpose

           

          Nicolas De Loof added a comment - weakcamel this is uncertain so far, as this issue is more about balancing what we'd like to get and what is feasible . Typically we'd like to be able to run arbitrary docker image but  jenkins require a JVM for it's agent we'd like to be able to share workspace between steps running in containers,  but then the uid/gid issue will come back  etc ... always with  but xx at some point IMHO we need to define a fresh new pipeline approach for docker usage that would not try to hack docker container. Working on a proposal on this purpose  

          Waldek M added a comment -

          ndeloof sure, I hear what you're saying. There's a lot of features Jenkins users take for granted (which they wouldn't like to use) and general design of agents stand in the way too.

          I've had a couple of discussions with my colleagues on the subject of sharing files between a host and a container and the bottom line was: the alternatives may be inconvenient (or complicated, or come  with performance trade-off), but using a simple volume mount just isn't clean and versatile enough and

          IMO the only clean way to transfer files back and forth is some sort of RPC (as nicolaw has suggested in JENKINS-47026), be it directly via SSH/SFTP/what have you or e.g. https://github.com/vieux/docker-volume-sshfs..] otherwise the design is always going to make assumptions on the image or Docker host setup. Just my £0.02.

          Waldek M added a comment - ndeloof sure, I hear what you're saying. There's a lot of features Jenkins users take for granted (which they wouldn't like to use) and general design of agents stand in the way too. I've had a couple of discussions with my colleagues on the subject of sharing files between a host and a container and the bottom line was: the alternatives may be inconvenient (or complicated, or come  with performance trade-off), but using a simple volume mount just isn't clean and versatile enough and IMO the only clean way to transfer files back and forth is some sort of RPC (as nicolaw has suggested in JENKINS-47026 ), be it directly via SSH/SFTP/what have you or e.g. https://github.com/vieux/docker-volume-sshfs. .] otherwise the design is  always going to make assumptions on the image or Docker host setup. Just my £0.02.

          weakcamel yes indeed, I'd prefer we enforce use of pipeline `squash` for file sharing, and get rid of the uid/gid ownership constraint

          Nicolas De Loof added a comment - weakcamel yes indeed, I'd prefer we enforce use of pipeline `squash` for file sharing, and get rid of the uid/gid ownership constraint

          Some update on this issue:

          we'd like to introduce some "version:2" in declarative pipeline so changing behaviour to this new agent design would be opt-in only and we wouldn't break existing pipelines.

          docker-plugin do support `dockerNode` pipeline DSL keyword to provision a full jenkins node within a pipeline. This sounds like a nice candidate for docker agent implementation in declarative, as we would then get a full agent to fully honour Jenkins APIs and expectations (wrappers, env, path, etc).

          A major issue to remain is the way docker.inside to let one access workspace from higher level agent. This introduce filesystem permission issues, so forced user to run in container. To get rid of this, safer option seems imho to not keep this behaviour for nested agents, and only support use of `stash` to share files between agents.

          Nicolas De Loof added a comment - Some update on this issue: we'd like to introduce some "version:2" in declarative pipeline so changing behaviour to this new agent design would be opt-in only and we wouldn't break existing pipelines. docker-plugin do support `dockerNode` pipeline DSL keyword to provision a full jenkins node within a pipeline. This sounds like a nice candidate for docker agent implementation in declarative, as we would then get a full agent to fully honour Jenkins APIs and expectations (wrappers, env, path, etc). A major issue to remain is the way docker.inside to let one access workspace from higher level agent. This introduce filesystem permission issues, so forced user to run in container. To get rid of this, safer option seems imho to not keep this behaviour for nested agents, and only support use of `stash` to share files between agents.

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          pom.xml
          src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy
          src/main/java/io/jenkins/docker/pipeline/DockerAgent.java
          src/main/java/io/jenkins/docker/pipeline/DockerAgentScript.groovy
          http://jenkins-ci.org/commit/docker-plugin/a8240b9a147153a9c2c772bcc8df4acdd8798d19
          Log:
          JENKINS-48050 declarative « container » agent

          Compare: https://github.com/jenkinsci/docker-plugin/compare/6acd9e848f61^...a8240b9a1471

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy src/main/java/io/jenkins/docker/pipeline/DockerAgent.java src/main/java/io/jenkins/docker/pipeline/DockerAgentScript.groovy http://jenkins-ci.org/commit/docker-plugin/a8240b9a147153a9c2c772bcc8df4acdd8798d19 Log: JENKINS-48050 declarative « container » agent Compare: https://github.com/jenkinsci/docker-plugin/compare/6acd9e848f61 ^...a8240b9a1471

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          pom.xml
          src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy
          src/main/java/io/jenkins/docker/pipeline/DockerAgent.java
          http://jenkins-ci.org/commit/docker-plugin/65dce18da936fbeea34d89245595d085866dd725
          Log:
          JENKINS-48050 declarative « container » agent

          Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy src/main/java/io/jenkins/docker/pipeline/DockerAgent.java http://jenkins-ci.org/commit/docker-plugin/65dce18da936fbeea34d89245595d085866dd725 Log: JENKINS-48050 declarative « container » agent Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          pom.xml
          src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy
          src/main/java/io/jenkins/docker/pipeline/DockerAgent.java
          http://jenkins-ci.org/commit/docker-plugin/2e2c79fc3aba672e6a19d812d47ac55b2c7a418b
          Log:
          JENKINS-48050 declarative « container » agent

          Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy src/main/java/io/jenkins/docker/pipeline/DockerAgent.java http://jenkins-ci.org/commit/docker-plugin/2e2c79fc3aba672e6a19d812d47ac55b2c7a418b Log: JENKINS-48050 declarative « container » agent Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          pom.xml
          src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy
          src/main/java/io/jenkins/docker/pipeline/DockerAgent.java
          http://jenkins-ci.org/commit/docker-plugin/3cea7e1a5c49d069be4aeea1b027210101ce4bac
          Log:
          JENKINS-48050 declarative « container » agent

          Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: pom.xml src/main/groovy/io/jenkins/docker/pipeline/DockerAgentScript.groovy src/main/java/io/jenkins/docker/pipeline/DockerAgent.java http://jenkins-ci.org/commit/docker-plugin/3cea7e1a5c49d069be4aeea1b027210101ce4bac Log: JENKINS-48050 declarative « container » agent Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

          Jesse Glick added a comment -

          Alternate simpler approach: docker PR 681

          Jesse Glick added a comment - Alternate simpler approach: docker PR 681

          Liam Newman added a comment -

          Liam Newman added a comment - I've added a link to some work done for this https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/255

            Unassigned Unassigned
            jamesdumay James Dumay
            Votes:
            14 Vote for this issue
            Watchers:
            41 Start watching this issue

              Created:
              Updated: