optimize getOldestRequestTime

[dustin]

The default prioritizeBuilders calls getOldestRequestTime for every configured builder. The net result is to fetch the entire buildrequests table, bit by bit, through a lot of queries.

This could probably be done a lot better at the prioritizeBuilders level, rather than calling into the guts of each builder. Catlee has some code to accomplish this already, which should be merged.

[aivarsk]

Can you provide a link to Catlee's code?

This kills performance of our buildmaster. We have 35 builders and each getOldestRequestTime SELECT takes 0.01-0.07 seconds (buildrequests has 125,400 records and buildsets has 14,461 records). 10 builders have about 650 buildrequests each day (nightly build for 600 "projects") and others about 50 buildrequests per day.

Tables had 1,344,969 and 165,365 records before and queries often showed up in logfiles ("Long query..." with 2 and more seconds). Not sure why tables were not cleaned up.

[dustin]

Catlee's patch is for 0.8.2, and won't apply anymore. After a quick look through the code (e.g.,  http://hg.mozilla.org/build/buildbot/annotate/d73b422653de/master/buildbot/process/builder.py), I can't find it. Maybe it never landed?

0.8.4 and 0.8.5 do quite a bit better at this -- only builders which actually have outstanding build requests are sorted, rather than all builders. Are there a lot of unclaimed build requests that you're finding? Could you define a less computationally-intensive prioritizeBuilders function?

[aivarsk]

Thanks for the idea about custom prioritizeBuilders function. I'm not familiar with buildbot DB structure so I can't answer about unclaimed builds. But I will take a closer look because state.sqlite keeps growing (it was 0.5Gb before last manual cleanup)

[dustin]

This is improved in recent versions, and will be further improved in nine.