Metrics framework

[catlee]

There are various moving parts to the buildbot master, and it would be nice to know how well they're performing on a running master.

Some examples:

• Cache hit rates
• Time spent in scheduler loop
• Time spent in builder loop
• # builds or steps starting per unit time

[dustin]

Users should be able to add their own metrics, too.

There should be a simple interface for querying these metrics, e.g., self.master.metrics.getMetric(metricname)

Metrics will need to be able to sustain themselves over master restarts and reconfigs. I've been thinking of making scheduler_state into a more generic saved_state (owner, key, value) store, which would allow this sort of state storage without new db tables.

Are metrics per-master, or clusterwide?

[catlee]

Metrics are per-master.

Also, I'm not sure if saving metrics across restarts is required. If the metrics interface has some notion of 'uptime', you can tell when your metrics are reset. It should be up to whoever is consuming the metrics to sum things properly.

[dustin]

dup #1945