Opened 4 years ago

Last modified 4 years ago

#3360 new defect

"Never saw reconfiguration finish." due to log rotation

Reported by: dcoshea Owned by:
Priority: minor Milestone: undecided
Version: master Keywords:
Cc:

Description

I got a "Never saw reconfiguration finish." message which appeared suspiciously quickly:

$ buildbot reconfig
sending SIGHUP to process 4535
[...] [-] beginning configuration update
[...] [-] Loading configuration from [...]
[...] [-] [a message my master.cfg "print"s]
[...] [-] [a message my master.cfg "print"s]
[...] [-] removing scheduler [...]
[... 13 more messages of the above form ...]
Never saw reconfiguration finish.

I checked and twistd.log got rotated in the middle of the reconfiguration - the messages that 'buildbot reconfig' output were now in twistd.log.1, and the rest of the output was in twistd.log:

[... 100 lines of other messages ...]
[...] [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x7f1cdc063320>
[...] [-] Setting up http.log rotating 10 files of 10000000 bytes each
[...] [-] configuration update complete

We have 53 schedulers, so lots of messages during reconfiguration, which I guess increases the opportunity to hit this bug. I don't think I've ever seen it before though.

Note that I'm using Buildbot from master at commit ID 9cd7f9a, which is a little over 2 years old. I'm hoping to upgrade to 0.8.12 soon.

Change History (1)

comment:1 Changed 4 years ago by dustin

This is still more-or-less a problem in newer versions. The log-following is basically a hack.

If you want to start buildbot in automation, it may be better to use twistd -noy path/to/buildbot.tac instead.

Ideally we'd transition to some kind of IPC or other indication of a successful startup.

Note: See TracTickets for help on using tickets.