buildbot throws lots of Exceptions when a Build's logs disappear before it finishes

[bhearsum]

We see this case whenever a slave gets hung mid-job and we don't notice until after a something has cleaned up its master-side logs. There's a few Exceptions I've seen like this. Here's a couple:

        Traceback (most recent call last):
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks 
            self._runCallbacks()
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
            self._startRunCallbacks(result)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> --- 
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 1064, in <lambda>
            d.addCallback(lambda res: self.createSummary(cmd.logs['stdio']))
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbotcustom/steps/unittest.py", line 808, in createSummary
            
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbotcustom/steps/unittest.py", line 144, in summarizeLogJetpacktests
            for line in log.readlines():
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 400, in readlines
            alltext = "".join(self.getChunks([channel], onlyText=True))
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 339, in getChunks
            f = self.getFile()
          File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 317, in getFile
            return open(self.getFilename(), "r")
        exceptions.IOError: [Errno 2] No such file or directory: './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'

2011-03-13 19:51:48-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 318, in callback
	    self._startRunCallbacks(result)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
	    self._runCallbacks()
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	--- <exception caught here> ---
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 885, in failed
	    self.step_status.stepFinished(EXCEPTION)
	  File "/builds/buildbot/tests_master6/lib/python2.6/site-packages/buildbot-0.8.2_hg_a63f22816750_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1047, in stepFinished
	    if os.path.getsize(loog.getFilename()) > logCompressionLimit:
	  File "/builds/buildbot/tests_master6/lib/python2.6/genericpath.py", line 49, in getsize
	    return os.stat(filename).st_size
	exceptions.OSError: [Errno 2] No such file or directory: './mozilla-central_leopard-o-debug_test-jetpack/343-log-jetpack-stdio'

These sort of things need to get reported somewhere, but raw Exceptions in the twistd.log don't seem like the right place. Ideally, errors like this should make their way to Status plugins, but I don't think that's possible yet. For now, maybe it's best to report these in a nicer way in twistd.log.

[dustin]

In theory, we don't flush active builds, so fixing that is probably the appropriate response.

[dustin]

It turns out ( https://bugzilla.mozilla.org/show_bug.cgi?id=641788) that this was because of a process external to Buildbot that's deleting pickles. So, um, "don't do that" :)