Opened 7 years ago

Last modified 4 years ago

#1688 assigned enhancement

there should be some way to retry builds that fail in setupBuild without submitting new Changes.

Reported by: bhearsum Owned by:
Priority: major Milestone: 0.9.+
Version: 0.8.2 Keywords:
Cc: catlee, dustin

Description (last modified by tardyp)

We hit an issue on a master recently where a reconfig failed part way through and caused a bunch of our custom code to have bad references in it. In turn, this caused a great number of builds to fail with a setupBuild exception. Even when a setupBuild exception occurs for another reason it really sucks that the builds never start and are never retried. I think that Buildbot should either retry them automatically are make it a lot easier to retry them.

Because the builds never started we wouldn't be able to make use of the RETRY status to make this happen. I was chatting with Catlee about this and he suggested that we might be able to accomplish this by running newBuild() in a try/except block and resetting claimed_at/claimed_by if it fails, possibly after waiting some period of time.

Change History (8)

comment:1 Changed 7 years ago by dustin

  • Milestone changed from undecided to 0.8.+
  • Type changed from undecided to enhancement

I'm curious what's going wrong in setupBuild, and in particular what problems do you see that might be remedied by a retry? If there's a transient error in setupBuild the buildbot source, we (Buildbotters) should fix that. If the error is in Mozilla code, the other we (with my Mozilla hat on) should fix that by retrying whatever action is producing the error.

Otherwise, if something goes wrong enough to trigger a failure in setupBuild, I can't see it being anything but purple - which it's not right now, so I'll leave this bug open to fix that problem.

comment:2 Changed 6 years ago by dustin

  • Cc changed from catlee,dustin to catlee, dustin

Looking back, I think catlee's suggestion was correct - if setupBuild fails, the claim should be retired and the build request will be re-assigned (well, attempted) again.

comment:3 Changed 5 years ago by tom.prince

  • Milestone changed from 0.8.+ to 0.8.8

comment:4 Changed 5 years ago by tom.prince

  • Owner set to jaredgrubb
  • Status changed from new to assigned

comment:5 Changed 5 years ago by dustin

  • Milestone changed from 0.8.8 to 0.9.0

This is something we should revisit in the updated build-claiming code in 0.9.0. Builds should either be started or build requests unclaimed, in any circumstance.

comment:6 Changed 4 years ago by tardyp

  • Description modified (diff)
  • Owner changed from jaredgrubb to dustin

reassign to dustin, as jared is not really working on it anymore

comment:7 Changed 4 years ago by dustin

  • Milestone changed from 0.9.0 to 0.9.+

comment:8 Changed 4 years ago by dustin

  • Owner dustin deleted
Note: See TracTickets for help on using tickets.