Opened 6 years ago

Last modified 5 years ago

#2224 new enhancement

Support MySQL partitioning

Reported by: dustin Owned by:
Priority: major Milestone: 0.9.+
Version: 0.8.2 Keywords: database
Cc:

Description

A big Buildbot install dumps massive amounts of data into the DB, and its "normal" pruning process is just DELETE statements, which don't necessarily free up space, and at any rate can leave tables badly fragmented.

A DBA will tell you that the fix is to partition the data by time, and simply drop partitions when they are no longer needed. No fragmentation, no delays on pruning, no muss, no fuss.

I think that this is *primarily* something that would be done by a skilled DBA without modification of Buildbot itself, but let's figure out if Buildbot can make this easier or if there are other modifications that could help.

Change History (2)

comment:1 Changed 6 years ago by dustin

From what I can gather, this wouldn't require any changes to buildbot. At best, we could provide a contrib script to set this up (alter tables, etc.).

comment:2 Changed 5 years ago by dustin

This done here for Mozpool:

https://github.com/mozilla/mozpool/blob/master/sql/schema.sql

I think that the Buildbot support for this would basically involve changing the SQL used to prune builds, logs, etc. Instead of running a 'DELETE' query, Buildbot would invoke a partition-updating function. We could supply that function in a contrib script.

Nine currently doesn't prune anything, so this will need to wait a bit.

Note: See TracTickets for help on using tickets.