Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#2115 closed defect (fixed)

Periodic builder stops being assigned to slave

Reported by: ajdavis Owned by:
Priority: major Milestone: 0.8.4
Version: 0.8.4p1 Keywords:
Cc:

Description

Maybe related to #85: I have a single master and single slave, and 4 once-per-minute periodic builds. The master and slave have been running for a few days, and at some point ONE of the periodic builds stopped being assigned (I think that's the problem), while the other 3 builds are continuing once per minute as normal. Pending builds for the builder (called "send_confirmation_emails_and_smses") have stacked up so that there are thousands pending.

Attachments (6)

Screen shot 2011-09-27 at 2.20.46 PM.png (32.4 KB) - added by ajdavis 7 years ago.
builders
Screen shot 2011-09-27 at 2.22.28 PM.png (44.7 KB) - added by ajdavis 7 years ago.
buildslaves
Screen shot 2011-09-27 at 2.23.22 PM.png (64.9 KB) - added by ajdavis 7 years ago.
last successful build of the stuck builder
Screen shot 2011-09-27 at 2.24.22 PM.png (185.9 KB) - added by ajdavis 7 years ago.
builder with thousands of pending builds
twistd.log (34.9 KB) - added by ajdavis 7 years ago.
relevant chunk of master/twistd.log
master.cfg (5.5 KB) - added by ajdavis 7 years ago.

Download all attachments as: .zip

Change History (9)

Changed 7 years ago by ajdavis

builders

Changed 7 years ago by ajdavis

buildslaves

Changed 7 years ago by ajdavis

last successful build of the stuck builder

Changed 7 years ago by ajdavis

builder with thousands of pending builds

Changed 7 years ago by ajdavis

relevant chunk of master/twistd.log

Changed 7 years ago by ajdavis

comment:1 Changed 7 years ago by ajdavis

sudo /etc/init.d/buildbot restart fixes the problem for now. I may write a cron job to restart buildbot nightly. Who watches the watchmen?

comment:2 Changed 7 years ago by dustin

  • Milestone changed from undecided to 0.8.4
  • Resolution set to fixed
  • Status changed from new to closed

I'm missing how that chunk of twistd.log is relevant? It looks like it's *before* the last successful build?

0.8.4p1 had some problems with dropping build requests. Try 0.8.4p2, or even 0.8.5? Re-open with a changed version field if it's still failing the same way.

comment:3 Changed 7 years ago by ajdavis

Thanks, I've upgraded to 0.8.5. We'll see.

Note: See TracTickets for help on using tickets.