buildbot waterfall page shows error after reconfig

[pwp]

buildbot waterfall page shows the following errors after reconfig

---------
web.Server Traceback (most recent call last):
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'getStatus'

/usr/lib/python2.6/dist-packages/twisted/web/server.py, line 125 in process
123    try:
124      resrc = self.site.getResourceFor(self)
125      self.render(resrc)
126    except:
Self
site	
buildbot.status.web.baseweb.RotateLogSite instance @ 0xa0165a8
<buildbot.status.web.baseweb.RotateLogSite instance at 0xa0165a8>
Locals
resrc	<buildbot.status.web.waterfall.WaterfallStatusResource object at 0xb01bf50>
self	
twisted.web.server.Request instance @ 0x8e084d0
<GET /waterfall?reload=900 HTTP/1.1>
/usr/lib/python2.6/dist-packages/twisted/web/server.py, line 132 in render
130  def render(self, resrc):
131    try:
132      body = resrc.render(self)
133    except UnsupportedMethod, e:
Locals
resrc	<buildbot.status.web.waterfall.WaterfallStatusResource object at 0xb01bf50>
self	
twisted.web.server.Request instance @ 0x8e084d0
<GET /waterfall?reload=900 HTTP/1.1>
Globals
UnsupportedMethod	<class 'twisted.web.error.UnsupportedMethod'>
/usr/local/lib/python2.6/dist-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/status/web/base.py, line 324 in render
322      return ''
323
324    ctx = self.getContext(request)
325
Locals
self	<buildbot.status.web.waterfall.WaterfallStatusResource object at 0xb01bf50>
request	
twisted.web.server.Request instance @ 0x8e084d0
<GET /waterfall?reload=900 HTTP/1.1>
/usr/local/lib/python2.6/dist-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/status/web/base.py, line 196 in getContext
194class ContextMixin(AccessorMixin):
195  def getContext(self, request):
196    status = self.getStatus(request)
197    rootpath = path_to_root(request)
Locals
self	<buildbot.status.web.waterfall.WaterfallStatusResource object at 0xb01bf50>
request	
twisted.web.server.Request instance @ 0x8e084d0
<GET /waterfall?reload=900 HTTP/1.1>
Globals
ContextMixin	<class 'buildbot.status.web.base.ContextMixin'>
AccessorMixin	<class 'buildbot.status.web.base.AccessorMixin'>
path_to_root	function path_to_root in file /usr/local/lib/python2.6/dist-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/status/web/base.py at line 111
/usr/local/lib/python2.6/dist-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/status/web/base.py, line 182 in getStatus
180class AccessorMixin(object):
181  def getStatus(self, request):
182    return request.site.buildbot_service.getStatus()
183
Locals
self	<buildbot.status.web.waterfall.WaterfallStatusResource object at 0xb01bf50>
request	
twisted.web.server.Request instance @ 0x8e084d0
<GET /waterfall?reload=900 HTTP/1.1>
Globals
AccessorMixin	<class 'buildbot.status.web.base.AccessorMixin'>
/usr/local/lib/python2.6/dist-packages/buildbot-0.8.6p1-py2.6.egg/buildbot/status/web/baseweb.py, line 498 in getStatus
496
497  def getStatus(self):
498    return self.master.getStatus()
499
Self
master	None
Locals
self	
buildbot.status.web.baseweb.WebStatus instance @ 0x52ea830
<WebStatus on port tcp:8010 at 0x52ea830>
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'getStatus'

---

Reproduce steps:

1. open waterfall page and leave it open
2. buildbot reconfig
3. refresh waterfall page

2 out of 3 times you will see these errors. The only way is to close browser and reopen the page. 0.8.4 does not have this issue after using for long time.

[pwp]

error page

[tom.prince]

Does moving the

        service.MultiService.setServiceParent(self, parent)

line from the beginning of buildbot/status/web/baseweb.py:WebStatus.setServiceParent to the end fix the issue?

[tom.prince]

 https://github.com/tomprince/buildbot/commit/bug2301

[pwp]

You mean to move this line the end of the function setServiceParent like?

        self.setupSite()
        service.MultiService.setServiceParent(self, parent)

[tom.prince]

Yes.

[dustin]

That seems unlikely to help -- site.buildbot_service is already set toward the end of setServiceParent, and that value is used in the second-to-last stack frame.

I can reproduce while reconfig'ing every 0.5s and clicking around webStatus:

2012-05-19 19:01:14-0500 [-] configuration update complete
2012-05-19 19:01:15-0500 [HTTPChannel,0,172.16.1.20] Unhandled Error
        Traceback (most recent call last):
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/http.py", line 1349, in dataReceived
            finishCallback(data[contentLength:])
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/http.py", line 1563, in _finishRequestBody
            self.allContentReceived()
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/http.py", line 1618, in allContentReceived
            req.requestReceived(command, path, version)
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/http.py", line 773, in requestReceived
            self.process()
        --- <exception caught here> ---
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/server.py", line 131, in process
            resrc = self.site.getResourceFor(self)
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/server.py", line 562, in getResourceFor
            return resource.getChildForRequest(self.resource, request)
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/resource.py", line 65, in getChildForRequest
            resource = resource.getChildWithDefault(pathElement, request)
          File "/home/dustin/code/buildbot/t/buildbot/sand27/lib/python2.7/site-packages/twisted/web/resource.py", line 169, in getChildWithDefault
            return self.getChild(path, request)
          File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/status/web/builder.py", line 556, in getChild
            s = self.getStatus(req)
          File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/status/web/base.py", line 182, in getStatus
            return request.site.buildbot_service.getStatus()
          File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/status/web/baseweb.py", line 498, in getStatus
            return self.master.getStatus()
        exceptions.AttributeError: 'NoneType' object has no attribute 'getStatus'

2012-05-19 19:01:15-0500 [-] beginning configuration update

but I think that was just getting lucky. I can't reproduce with the suggested patch, but I also can't reproduce a second time *with* the suggested patch..

[dustin]

This seems to be caused by an HTTP request at the wrong point in the reconfig process.

I've now successfully reproduced both with and without the patch in comment 3. So something else is afoot here.

[Tom Prince]

Fix race condition during web status reconfig.

Service.setServiceParent starts the current service, if the parent is already started. Thus, the current service needs to be ready to run when we call it.

Fixes #2301.

Changeset: 34a3f6d9a6121b860d36f7c5b80f5bb96a251119

[Dustin J. Mitchell]

Don't reset webstatus.master to None when it is replaced

This is an awful hack, but re-creating WebStatus? on every reconfig is also a bad idea. Fixes #2301.

Changeset: cc5b2ed5a952250846d98b06617dbf346bbc9998

[Tom Prince]

Fix race condition during web status reconfig.

Service.setServiceParent starts the current service, if the parent is already started. Thus, the current service needs to be ready to run when we call it.

Fixes #2301.

Changeset: 34a3f6d9a6121b860d36f7c5b80f5bb96a251119

[Dustin J. Mitchell]

Don't reset webstatus.master to None when it is replaced

This is an awful hack, but re-creating WebStatus? on every reconfig is also a bad idea. Fixes #2301.

Changeset: cc5b2ed5a952250846d98b06617dbf346bbc9998

[pitrou]

For the record, the patches seem to have fixed the issues which had appeared on  http://buildbot.python.org after upgrading to 0.8.6p1.