Reconfig after adding step results in buildException

[tmidd]

I added an extra step to a builder and then reconfig'ed the master. The step I added was automatically named MasterShellCommand_2 which pushed the previous owner of that name to MasterShellCommand_3 (which didn't exist previously). There were no builds running on the builder when I did the reconfig.

The next time the builder finished it threw the following exception and failed to "end" correctly - that is, subsequent builds are pending even though there is no active build. Restarting the master seems to have fixed the problem.

2012-03-22 14:22:25+1300 [-]  setting expectations for next time
2012-03-22 14:22:25+1300 [-] <Build balance-incr>.buildException
2012-03-22 14:22:25+1300 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 396, in _stepDone
            return self.startNextStep()
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 386, in startNextStep
            d.addCallback(self._stepDone, s)
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 301, in addCallback
            callbackKeywords=kw)
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 290, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 396, in _stepDone
            return self.startNextStep()
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 383, in startNextStep
            return self.allStepsDone()
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 486, in allStepsDone
            return self.buildFinished(text, self.result)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 517, in buildFinished
            self.builder.setExpectations(self.progress)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/builder.py", line 450, in setExpectations
            self.expectations.update(progress)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/status/progress.py", line 295, in update
            old = self.times[name]
        exceptions.KeyError: 'MasterShellCommand_3'

It looks like it's trying to find the new step in a dict that wasn't updated correctly by the reconfig, although I haven't had a proper look through the code yet to confirm exactly what's going wrong.

[tmidd]

Just noticed this exception is followed by another a minute or so later:

2012-03-22 14:23:39+1300 [-] Unhandled error in Deferred:
2012-03-22 14:23:39+1300 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 396, in _stepDone
            return self.startNextStep()
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 387, in startNextStep
            d.addErrback(self.buildException)
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 312, in addErrback
            errbackKeywords=kw)
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 290, in addCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 491, in buildException
            self.buildFinished(["build", "exception"], EXCEPTION)
          File "/usr/local/lib/python2.7/site-packages/buildbot-0.8.6-py2.7.egg/buildbot/process/build.py", line 507, in buildFinished
            self.remote.dontNotifyOnDisconnect(self.lostRemote)
          File "/usr/local/lib/python2.7/site-packages/Twisted-12.0.0-py2.7-linux-x86_64.egg/twisted/spread/pb.py", line 306, in dontNotifyOnDisconnect
            self.disconnectCallbacks.remove(callback)
        exceptions.ValueError: list.remove(x): x not in list

[Tom Prince]

Fix crashes when adding new steps in reconfig.

Fixes #2252.

Changeset: b915a48d7ef30da0dbb80397911737e53473ce49