Ticket #2115 (closed defect: fixed)

Opened 20 months ago

Last modified 20 months ago

Periodic builder stops being assigned to slave

Reported by: ajdavis Owned by:
Priority: major Milestone: 0.8.4
Version: 0.8.4p1 Keywords:
Cc:

Description

Maybe related to #85: I have a single master and single slave, and 4 once-per-minute periodic builds. The master and slave have been running for a few days, and at some point ONE of the periodic builds stopped being assigned (I think that's the problem), while the other 3 builds are continuing once per minute as normal. Pending builds for the builder (called "send_confirmation_emails_and_smses") have stacked up so that there are thousands pending.

Attachments

Screen shot 2011-09-27 at 2.20.46 PM.png Download (32.4 KB) - added by ajdavis 20 months ago.
builders
Screen shot 2011-09-27 at 2.22.28 PM.png Download (44.7 KB) - added by ajdavis 20 months ago.
buildslaves
Screen shot 2011-09-27 at 2.23.22 PM.png Download (64.9 KB) - added by ajdavis 20 months ago.
last successful build of the stuck builder
Screen shot 2011-09-27 at 2.24.22 PM.png Download (185.9 KB) - added by ajdavis 20 months ago.
builder with thousands of pending builds
twistd.log Download (34.9 KB) - added by ajdavis 20 months ago.
relevant chunk of master/twistd.log
master.cfg Download (5.5 KB) - added by ajdavis 20 months ago.

Change History

Changed 20 months ago by ajdavis

builders

Changed 20 months ago by ajdavis

buildslaves

Changed 20 months ago by ajdavis

last successful build of the stuck builder

Changed 20 months ago by ajdavis

builder with thousands of pending builds

Changed 20 months ago by ajdavis

relevant chunk of master/twistd.log

Changed 20 months ago by ajdavis

comment:1 Changed 20 months ago by ajdavis

sudo /etc/init.d/buildbot restart fixes the problem for now. I may write a cron job to restart buildbot nightly. Who watches the watchmen?

comment:2 Changed 20 months ago by dustin

  • Status changed from new to closed
  • Resolution set to fixed
  • Milestone changed from undecided to 0.8.4

I'm missing how that chunk of twistd.log is relevant? It looks like it's *before* the last successful build?

0.8.4p1 had some problems with dropping build requests. Try 0.8.4p2, or even 0.8.5? Re-open with a changed version field if it's still failing the same way.

comment:3 Changed 20 months ago by ajdavis

Thanks, I've upgraded to 0.8.5. We'll see.

Note: See TracTickets for help on using tickets.