make buildmaster a lot faster

[dustin]

I have about 40 builders, and they see a decent amount of action (a few hundred builds a month). Each builder has a lot of steps, and a lot of log output. My master's basedir is 2.3G! As a result, the buildmaster is very slow. I see this specifically in a few places:

• on startup, buildmaster loads the pickle for each builder, then (once all of the pickles are loaded) writes each pickle back out. For me, this takes 3-4 minutes
• on shutdown, buildmaster writes each builder's pickle to disk, taking another minute or so (and timing out my initscript)
• views of the grid and waterfall result in massive number of BuildStatus? pickles being loaded

I'm not sure what the best solution is, but I'm going to see what I can do. My darcs repo for the project is at darcs.r.igoro.us/buildbot/speedup.

[dustin]

see aso #24 -- using a db for the persistence

[warner]

Yeah, I think the DB status approach would speed up all three of those issues. Caching some of the intermediate tables used by the waterfall page would probably help a lot too. Make sure you take a look at the 'sqlify' branch mentioned in #24, I think the schema there is workable.

thanks!

[dustin]

OK, this turned out to be a lot harder than I thought, and now my tree is so divergent from trunk that it's never coming back. Oh well!