reconfig by graceful failover

[dustin]

A few alternatives to reconfiguration came up in IRC.

One particularly attractive model is to treat a reconfig as a time to spin up a new member of the master pool, then gracefully disconnect each slave on the "retiring" master when it finishes a build.

This obviously has a lot of prerequisites, but it would be a great model:

• fully db-based
• fully-baked master failover support
• abstracted status so you're not looking at two web statuses during the changeover

[dustin]

The new config interface actually does a bit better job with reconfigs, but there are still problems:

• reloading/pre-initializing Python from the config file (ugh, and we have some fixes)
• performance / memory consumption -- Buildbot can end up using a lot of memory (and whether this is Buildbot or custom modules is unclear - see the performance tag), and restarting the process is a great way to flush that.

So this is still necessary, but less soon.

[dustin]

see also #1880