jhoblitt, thanks for your thought. The bar of success I set for the system-config-dsl project is to pass the test of folks like you, so hearing from you is very valuable.

When I read your comment, I get the sense that you are trying to point out problems in the project, but all I actually see is the validation. I think this largely stems from the fact that you only see what's already there, and I have some additional pieces in my head that's not yet implemented.

So let me try to explain those additional pieces that are only in my head, and I'm really keen to hear from you again on whether that changes anything from your perspective.

If you leave out the "programmable" portion of DSL and focus on the rest that builds the tree structure, such as the example in README, I see a fairly simple data model that can be seen in just about any tree-capable syntax, such as JSON, YAML, and XML. This in my mind forms the first layer of the configuration file syntax, which you call "configuration as data". And I'm happy to give people those additional "language binding" just in case the particular curly brackets and parenthesis is off-putting to some people.

The programmability of DSL, such as ability to do variable expansions, loop, and so on is then a nice icing for people like me who wants to keep it DRY without relying on external templating tool. In other words, in this case I see "configuration as data" as a subset of "configuration as code."

Compared to "xstream object dump", this immediately wins a few things. One is that users are not constrained by the developers' choice of what information are colocated in one file vs what information are split into separate files. With system-config-dsl plugin, if you want to manage the authentication part but not the authorization part, there's no problem. Likewise you can use two separate files to manage two parts of configurations independently, even when the outcome ends up in the same XML file. You mention /etc/sudoers.d and /etc/yum.conf.d as a positive example, and I'm like, yep, system-config-dsl is solving that.

But ultimately I think the biggest benefit in this approach, compared to "xstream object dump", is that we can now statically generate semantic model of what properties exist on what sections top to bottom, across the entire plugin ecosystem. There is no need for users to be familar with Jenkins internal APIs, and no need for them to look up javadoc. For example, this semantic model can be used to generate a documentation comparable to that of Job DSL plugin, and it can be even used to generate offline semantic checker. You raise the point that one of the problems with "xstream object dump" is that it's difficult to reproduce & predict, and generally reason about. I feel like system-config-dsl is solving that exact problem here.

So in the end, from your conclusion below, in my mind the system-config-dsl plugin provides (1) well-defined, (2) data format, that (3) allows config to split across multiple files. And it solves the plugin installation use cases.

IMHO - the best filesystem based interface for a CM system is one that uses a well defined data/serialization format, allows the configuration to be split across multiple files in the now common foo.d style, and uses either extremely predictable file names (eg /etc/foo.conf), in that the same configuration values will always be in the same file, or the file name has no significance (eg., /etc/sudoers.d/, /etc/yum.conf.d)

So what am I missing here?

I'm just acknowledging here that I'm not making enough progress on this front despite my earlier hope.

I don't want my "claiming" this space to prevent someone else from carrying this flag forward.

Hi all,
I'd like to throw my 2 cents into this hat . In general, I've found the most effective way to configure Jenkins is to use the Script Console (link) and bootstrap Jenkins and plugins via gradle.

Bootstrapping Jenkins via gradle

Bootstrapping Jenkins via gradle provides something that is currently lacking in Jenkins documentation (which I'm slowly improving over time). The idea of idempotent configurations. That is, if I bootstrap my automation scripts next year they should still be guaranteed to work. The current state of most configuration management "solutions" I've seen is this isn't the case. However, Jenkins publishes the war file and all of it's plugins in Artifactory. This offers a great platform to guarantee idempotent configurations.

Here's an example of my scripts bootstrapping Jenkins and all plugin versions via gradle (link).

Advantages of this method:

• Guaranteed to bootstrap the same way next year as it does this year.
• Maven repository can be mirrored on-site in enterprises to ensure reliable bootstrapping.
• It's easier to spin up test environments with the same versions of Jenkins and plugins this way (even on my own laptop).
• Plugin updates and upgrades can be more safely quality tested and approved for upgrade in production.

"Jenkins State Machine"

When it comes to configuration management and an existing Jenkins installation I've always thought of what I call a "Jenkins State Machine". The Jenkins State Machine is essentially the following:

1. If Jenkins is not in shutdown mode, then configuration changes can occur in the Jenkins runtime.
2. If Jenkins is in shutdown mode, then don't make any configuration changes to Jenkins. i.e. any script which can change the runtime configuration should choose not to change if Jenkins is in this state.
3. If Jenkins is in shutdown mode and there are jobs running, then Jenkins should not be restarted. A "restart" configuration management script should always just choose to wait to allow running jobs to finish.
4. If Jenkins is in shutdown and there are no jobs running, then it is safe to restart Jenkins.

The above state machine can be relatively safely run, automatically, without affecting users too drastically. Something like this can be done without much human intervention as long as the state machine is adhered in all config management scripts.

The above state machine still poses minor problems. Any time jobs queue up during shutdown mode, the state of the queue is discarded on restart. The Jenkins job queue could be preserved when Jenkins is gracefully restarted or gracefully shut down.

Configure via Script Console

One of the most powerful things I've learned about Jenkins and the Script Console is that its runtime is itself the configuration. By updating the runtime, you're updating the configuration and can serialize that configuration to disk with a Jenkins.instance.save() or calling the save() method on other configuration objects.

Pros:

• The "Jenkins State Machine" can be adhered.
• Configuration is immediately updated and the operation is as safe as clicking "Save" in the web UI.
• No restart is required for trivial configuration updates via the Script Console. If the XML configuration on disk is modified then a Jenkins restart is required or executing "Reload configuration from disk". It's an intense and disruptive action. Configuring via Script Console scripts helps avoid this disruption.

Cons:

• Writing Script Console scripts can be labor intensive and is definitely a major automation cost.

In my job, my team and I have paid this up front automation cost for our Jenkins installation. It's amazing how quickly and safely we manage or recover our Jenkins. I plan to share our work in the future.

Some of my automation work around this is already in the wild. jervis_bootstrap.sh (link) shows off how to safely execute script console scripts in Jenkins 2.0 with authentication and CSRF protection enabled. In that same project, the scripts/ directory (link) shows several groovy scripts which are used to automatically configure Jenkins. This includes skipping the Jenkins 2.0 wizard, enabling CSRF protection, and configure other minor bits and bobs. This can be much more comprehensive.

Script Console Scripts are not only great for debugging but configuring Jenkins as well.

I have a video

I presented at a Jenkins Area Meetup and recorded it (link). I explain how I go about writing script console scripts in case others wish to pay this automation cost. After discussing far and wide within the Jenkins community, I'm fairly certain that this approach is unique and effective.

So I finally got around to coming back to this, after KK showed me this in May.

I'd say that both Joshua & KK are right-

The config "language" makes life much easier than dealing w/ XML. It's already safe for use in files. The key would be to get something akin to the script console, or an API, to both publish the current state as the DSL, and accept the DSL as an input.

One of the biggest issues out there is dealing w/ security setup, while still letting the CM system manage the tools. Binding a localhost only API that accepted the DSL commands - particularly partial files - would be a great help. And providing the current config as a DSL description would allow (in Puppet speak) a parser to be written for Jenkins as a package type, or as a resource type & provider.

This, plus the Pipeline DSL, makes things much easier.

The fact is: This kind of functionality, implemented as either a DSL, or as a series of REST API endpoints, would make life massively easier for setup & provisioning.

(The key though is to make it possible to access the API via a different, privileged connection that won't break when you turn on normal security)

In addition to the puppet-jenkins project, there is another initiative which seems to work towards the same goals:

https://docs.openstack.org/infra/jenkins-job-builder/

jhill, I don't believe Jenkins Job Builder (jjb) is in this same space. Pipeline, Job DSL plugin, and jjb all occupy a somewhat similar space. Whereas this ticket is a level deeper, about automating the management and configuration of the Jenkins environment itself, rather than the workloads that execute on top of it.

I've also been taking all of my experience in that script console configuration space and I've created a project to implement some of my ideas as an experiment.

Eventually it will be able to configure the master settings.  I'm brainstorming ways in which it can be dynamically compatible with plugins as well.  Check it out -> https://github.com/sandscape

Hi!

We have been doing Jenkins configuration through groovy scripts and already accumulated number of the scripts that used by our customers and us internally.

You can check them out here https://github.com/Praqma/JenkinsAsCodeReference/tree/master/dockerizeit/master

Next step for us would be to separate what from how and find a good way to distribute those scripts. Currently, we have a lot of forks of the repo and that is not the best way of distributing updates. We thought about wrapping them into a plugin. Thanks to rsandell who pointed us to config-dsl. ewel and I will try to put some work into config-dsl during the summer.

Is anyone working on it already? Or have wishes, requirements that we should take into account?

BR, Andrey

CloudBees and Praqma are joining hands on this effort, and ewel and ndeloof have started a new project that combines existing efforts. See the mission statement and the goal at JEP #201

Pardon my ignorance, is this the correct venue to comment on JEP #201 or is there a more appropriate location?