Opened 6 years ago

Last modified 4 years ago

#2218 new enhancement

Add support for declarative configuration.

Reported by: tom.prince Owned by:
Priority: patches-accepted Milestone: 1.0.+
Version: 0.8.5 Keywords: config
Cc: krf@…

Description (last modified by dustin)

Buildbot's configuration has historically been a Python file that Buildbot executes. The result of that execution is then used to set up the master. This assumes users can write valid Python code, and encourages people to do all manner of fragile or non-maintainable things. See #2218 for more.


This project will involve some careful thought about how users interact with Buildbot, and how we can maintain backward compatibility while providing users with a forward migration path. Once that's complete, the modifications to the code will be fairly minimal, and mostly limited to master/buildbot/ (with associated tests). However, the change will require alteration -- a rewrite, really -- of the entire "Configuration" section of the manual; this will be the most time-consuming (but critical) part of this project.


Attachments (1)

0001-Add-example-of-declarative-configuration-to-contrib.patch (18.9 KB) - added by dank 5 years ago.
sample implementation of declarative config

Download all attachments as: .zip

Change History (12)

comment:1 Changed 6 years ago by dustin

What does this mean?

comment:2 Changed 6 years ago by sanxiyn

I guess this means configuring with data, instead of code, say JSON.

I know you can do that already, but it would be nice to have built-in support if it can cover most cases, even if it does not cover all cases.

comment:3 Changed 6 years ago by dustin

The typical prescription for this is that we should ship with a simple, default configuration that parses such a declarative configuration. This shouldn't require any particular changes to Buildbot itself (other than in create-master): the "sample" master.cfg can parse the JSON (or whatever format is selected) and produce the resulting config. It might make more sense for master.cfg to call some parsing code embedded in Buildbot itself, e.g.

from buildbot.config import fromjson
BuildmasterConfig = fromjson.load("master_cfg.json")

I'm quite open to this idea. I think the most difficult part will not be the code, but the additional documentation -- suddenly every piece of configuration must be documented both the "easy" way and the "hard" (but more flexible) way.

comment:4 Changed 6 years ago by exarkun

Still better, particularly considering the relative effort compared to the interface in the above example, would be to just have BuildBot know how to load a master_cfg.json file (should JSON be selected as the right configuration format) and put the fromjson.load call somewhere in the implementation of BuildBot itself, not in the "configuration file" exposed to the admin for customization.

The former removes the attractive nuisance of a place to drop arbitrary, inscrutable Python code (an attractive nuisance place directly in front of every new buildbot user's face). The ability to use a Python-syntax master.cfg would be retained for quite some time, of course, to support old configurations and provide an escape hatch for functionality not yet supported by the non-Python configuration file. However, my ultimate goal would be to remove the unstructured dumping ground of master.cfg in favor of non-Python configuration and a more structured configuration plugin mechanism.

comment:5 Changed 6 years ago by dustin

Having the fromjson.load call in Buildbot makes sense. I also like that you've separated plugins (which to me means steps, schedulers, status receivers, and so on) from configuration. Lots of Buildbot users do dynamic configuration generation -- it's even done in the metabuildbot configuration -- but that can be handled by generating JSON.

This would have the added advantage of getting away from the pains of the existing format:

  • real, functional objects are instantiated in the config (e.g., schedulers)
  • lots of backward compatiblity
  • different case requirements
  • import paths used in configs live forever
  • ..I could think of more..

In fact, this wouldn't be horribly difficult to accomplish with the new MasterConfig support.

Which brings us to the painful but all-too-common question: who's up for implementing this? I'll start by adding it to ProjectIdeas.

comment:6 Changed 5 years ago by dank

(Neither here nor there, but I just noticed that recommends using an earlier try at declarative configuration. I suppose somebody should write a better developerworks article sometime.)

comment:7 Changed 5 years ago by dustin

  • Description modified (diff)
  • Type changed from enhancement to project-idea

Previous description:

<@exarkun> That's why you also have plugins
<@exarkun> It's just about actually separating your code from your configuration
<@exarkun> Your wacky auto-generated list of slave settings is fine, but put it in a module, unit test it (hey, document it! distribute it!)
<@exarkun> and then in your _configuration_ say "and please use wacky auto-generate slave plugin"
Last edited 5 years ago by dustin (previous) (diff)

comment:8 Changed 5 years ago by krf

  • Cc krf@… added

Changed 5 years ago by dank

sample implementation of declarative config

comment:9 Changed 5 years ago by dank

How's the attached sample look? It's a slightly cleaned up version of the one from my Big Fat Buildbot example; I've been using it in anger for several months now. If that looks interesting, I could add a section to the tutorial showing how to use it.

comment:10 Changed 5 years ago by dustin

  • Description modified (diff)

I like this a lot. I don't think it quite closes this bug, but it'll make a good addition to the project.

comment:11 Changed 4 years ago by dustin

  • Priority changed from major to patches-accepted
  • Type changed from project-idea to enhancement

I'm moving this out of Project-Ideas, because I don't think it makes a great GSoC project -- too much design and room for bikeshedding. It's still a good idea, though, and if it becomes much more explicit -- design complete, with just the implementation to finish -- then it would make a good GSoC project.

Note: See TracTickets for help on using tickets.