Http log rotation triggers exceptions.WindowsError on master when multiple web ports are present

[dustin]

Caemyr encountered a situation where, once the httpd log reached 1M, the twistd.log started to fill up with lengthy exceptions and everything ground to a halt.

Unfortunately, the exception got lost, but hopefully we can reproduce it.

[Caemyr]

Sample of two exception tracebacks from twisted log

[Caemyr]

Happens since update to 0.8.0. Master and two slaves on Windows 7 x64. All components in 32bit versions (x86):

Twisted: 10.0.0 Jinja: 2.4.1 Python: 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)]

This error is reproductible here. When http.log is filled up to the limit - 976 KB (1 000 135 bytes), web interface doesn't work properly anymore. Waterfall or build views are not refreshed, force build/cancel build buttons lead to error page (server interrupted connection).

Twisted log is filled by two kinds of tracebacks, listed in the file attached.

I found one workaround. It seems that stopping and restarting the master/slaves can in some cases (not 100%) fix this issue by rotating the log to the new file. Without the restart, this doesnt happen for sure.

[dustin]

I suspect that Twisted is getting itself stuck in a loop, which is why things get so slow. So the underlying problem is the file-in-use error.

I think that the workaround for 0.8.0 is just to disable http log rotation - see  http://djmitche.github.com/buildbot/docs/0.8.0/full.html#WebStatus-Configuration-Parameters Can you verify that this works?

I suspect that the longer-term fix involves a fix in Twisted..

[Caemyr]

This error happens only when there is more than one webport. With single port web rotation works fine. [20:32:45] <djmitche> so we really need to add a logFileBaseName=.. and verify that multiple WS's don't share the same name

Elevating to blocker as requested by Dustin. Changed ticket title.

[dustin]

Caemyr will be happy to test a patch for this. I'm leaving it in 0.8.2 for now, but it can get bumped without too much trouble.