builds automatically retried every hour

[dustin]

The buildrequest-claiming machinery marks the time a build is claimed, and considers it stale if the build is not claimed in RECLAIM_INTERVAL - by default, one hour. The comments indicate that the timestamp should be updated periodically, but this is not the case.

This was discovered at Mozilla in  https://bugzilla.mozilla.org/show_bug.cgi?id=608886.

[dustin]

There's something more subtle going on here - builder.py does, indeed, contain a TimerService that should re-claim all buildrequests every 10 minutes.

[catlee]

So the cause of this was an exception raised inside the reclaimAllBuilds call when DNS went away temporarily, the master couldn't resolve the hostname of the database server. The uncaught exception broke the TimerService?.

[dustin]

Fixed in [8ba6d68b52d1906da32704296157351804972dab]