Add support for declarative configuration.

[tom.prince]

Buildbot's configuration has historically been a Python file that Buildbot executes. The result of that execution is then used to set up the master. This assumes users can write valid Python code, and encourages people to do all manner of fragile or non-maintainable things. See #2218 for more.

scope

This project will involve some careful thought about how users interact with Buildbot, and how we can maintain backward compatibility while providing users with a forward migration path. Once that's complete, the modifications to the code will be fairly minimal, and mostly limited to master/buildbot/config.py (with associated tests). However, the change will require alteration -- a rewrite, really -- of the entire "Configuration" section of the manual; this will be the most time-consuming (but critical) part of this project.

examples

•  http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/config.json

[dustin]

What does this mean?

[sanxiyn]

I guess this means configuring with data, instead of code, say JSON.

I know you can do that already, but it would be nice to have built-in support if it can cover most cases, even if it does not cover all cases.

[dustin]

The typical prescription for this is that we should ship with a simple, default configuration that parses such a declarative configuration. This shouldn't require any particular changes to Buildbot itself (other than in create-master): the "sample" master.cfg can parse the JSON (or whatever format is selected) and produce the resulting config. It might make more sense for master.cfg to call some parsing code embedded in Buildbot itself, e.g.

from buildbot.config import fromjson
BuildmasterConfig = fromjson.load("master_cfg.json")

I'm quite open to this idea. I think the most difficult part will not be the code, but the additional documentation -- suddenly every piece of configuration must be documented both the "easy" way and the "hard" (but more flexible) way.

[exarkun]

Still better, particularly considering the relative effort compared to the interface in the above example, would be to just have BuildBot know how to load a master_cfg.json file (should JSON be selected as the right configuration format) and put the fromjson.load call somewhere in the implementation of BuildBot itself, not in the "configuration file" exposed to the admin for customization.

The former removes the attractive nuisance of a place to drop arbitrary, inscrutable Python code (an attractive nuisance place directly in front of every new buildbot user's face). The ability to use a Python-syntax master.cfg would be retained for quite some time, of course, to support old configurations and provide an escape hatch for functionality not yet supported by the non-Python configuration file. However, my ultimate goal would be to remove the unstructured dumping ground of master.cfg in favor of non-Python configuration and a more structured configuration plugin mechanism.

[dustin]

Having the fromjson.load call in Buildbot makes sense. I also like that you've separated plugins (which to me means steps, schedulers, status receivers, and so on) from configuration. Lots of Buildbot users do dynamic configuration generation -- it's even done in the metabuildbot configuration -- but that can be handled by generating JSON.

This would have the added advantage of getting away from the pains of the existing format:

• real, functional objects are instantiated in the config (e.g., schedulers)
• lots of backward compatiblity
• different case requirements
• import paths used in configs live forever
• ..I could think of more..

In fact, this wouldn't be horribly difficult to accomplish with the new MasterConfig support.

Which brings us to the painful but all-too-common question: who's up for implementing this? I'll start by adding it to ProjectIdeas.

[dank]

(Neither here nor there, but I just noticed that  http://www.ibm.com/developerworks/linux/library/l-buildbot/ recommends using  http://pypi.python.org/pypi/collective.buildbot an earlier try at declarative configuration. I suppose somebody should write a better developerworks article sometime.)

[dustin]

Previous description:

---

<@exarkun> That's why you also have plugins
<@exarkun> It's just about actually separating your code from your configuration
<@exarkun> Your wacky auto-generated list of slave settings is fine, but put it in a module, unit test it (hey, document it! distribute it!)
<@exarkun> and then in your _configuration_ say "and please use wacky auto-generate slave plugin"