Ticket #870 (new defect)

Opened 2 years ago

Last modified 3 months ago

Http log rotation triggers exceptions.WindowsError on master when multiple web ports are present

Reported by: dustin Owned by:
Priority: major Milestone: 0.8.+
Version: 0.8.0 Keywords: web, windows, sprint
Cc:

Description

Caemyr encountered a situation where, once the httpd log reached 1M, the twistd.log started to fill up with lengthy exceptions and everything ground to a halt.

Unfortunately, the exception got lost, but hopefully we can reproduce it.

Attachments

traceback.error.twisted.log Download (4.7 KB) - added by Caemyr 2 years ago.
Sample of two exception tracebacks from twisted log

Change History

Changed 2 years ago by Caemyr

Sample of two exception tracebacks from twisted log

comment:1 Changed 2 years ago by Caemyr

Happens since update to 0.8.0. Master and two slaves on Windows 7 x64. All components in 32bit versions (x86):

Twisted: 10.0.0 Jinja: 2.4.1 Python: 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)]

This error is reproductible here. When http.log is filled up to the limit - 976 KB (1 000 135 bytes), web interface doesn't work properly anymore. Waterfall or build views are not refreshed, force build/cancel build buttons lead to error page (server interrupted connection).

Twisted log is filled by two kinds of tracebacks, listed in the file attached.

I found one workaround. It seems that stopping and restarting the master/slaves can in some cases (not 100%) fix this issue by rotating the log to the new file. Without the restart, this doesnt happen for sure.

comment:2 Changed 2 years ago by dustin

  • Keywords web, windows added; web removed
  • Summary changed from http log rotation creates exception? to http log rotation triggers exceptions.WindowsError on master
  • Milestone changed from undecided to 0.8.0

I suspect that Twisted is getting itself stuck in a loop, which is why things get so slow. So the underlying problem is the file-in-use error.

I think that the workaround for 0.8.0 is just to disable http log rotation - see  http://djmitche.github.com/buildbot/docs/0.8.0/full.html#WebStatus-Configuration-Parameters Can you verify that this works?

I suspect that the longer-term fix involves a fix in Twisted..

comment:3 Changed 2 years ago by Caemyr

This error happens only when there is more than one webport. With single port web rotation works fine. [20:32:45] <djmitche> so we really need to add a logFileBaseName=.. and verify that multiple WS's don't share the same name

Elevating to blocker as requested by Dustin. Changed ticket title.

comment:4 Changed 2 years ago by Caemyr

  • Priority changed from critical to blocker

comment:5 Changed 2 years ago by Caemyr

  • Summary changed from http log rotation triggers exceptions.WindowsError on master to Http log rotation triggers exceptions.WindowsError on master when multiple web ports are present

comment:6 Changed 2 years ago by dustin

  • Milestone changed from 0.8.0 to 0.8.1

comment:7 Changed 22 months ago by dustin

Caemyr will be happy to test a patch for this. I'm leaving it in 0.8.2 for now, but it can get bumped without too much trouble.

comment:8 Changed 21 months ago by dustin

  • Milestone changed from 0.8.2 to 0.8.3

comment:9 Changed 18 months ago by ayust

  • Milestone changed from 0.8.3 to 0.8.+

comment:10 Changed 3 months ago by dustin

  • Keywords windows, sprint added; windows removed
  • Priority changed from blocker to major
Note: See TracTickets for help on using tickets.