Opened 4 years ago

Last modified 4 years ago

#3546 new defect

MasterShellCommand ends up in exception

Reported by: gracinet Owned by:
Priority: major Milestone: undecided
Version: master Keywords:
Cc:

Description

I've got a MasterShellCommand whose execution is ok (exit code is 0), yet whose result is EXCEPTION

There's no info on the Web UI, but here's what landed in twistd.log

2016-05-07 13:27:15+0200 [-] from an asynchronous method executed in an old-style step
        Traceback (most recent call last):
          File "/srv/buildbot9/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/srv/buildbot9/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1131, in _inlineCallbacks
            deferred.callback(getattr(e, "value", None))
          File "/srv/buildbot9/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
            self._startRunCallbacks(result)
          File "/srv/buildbot9/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/srv/buildbot9/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/srv/buildbot9/buildbot-git/master/buildbot/process/buildstep.py", line 169, in next
            self._catchup()
          File "/srv/buildbot9/buildbot-git/master/buildbot/process/buildstep.py", line 171, in _catchup
            self.step._start_unhandled_deferreds.append(d)
        exceptions.AttributeError: 'NoneType' object has no attribute 'append'
        
2016-05-07 13:27:15+0200 [-] Unhandled error in Deferred:

(and then the same error gets repeated several times). I've already checked that the 'old-style step' message is directly related to the Exception status

Precise rev (git describe): v0.9.0b8-229-g96f42bb

Change History (4)

comment:1 Changed 4 years ago by gracinet

I have a systematic reproduction on the current Git master, simply by putting a MasterShellCommand in the sample configuration master.cfg.sample.

This bug is not present in 0.9.0b8 The culprit is this (from buildbot.process.buildstep.py, in the run() method):

        finally:
            (...)
            self._start_deferred = None
            unhandled = self._start_unhandled_deferreds
            self._start_unhandled_deferreds = None
            self.realUpdateSummary()

        # Wait for any possibly-unhandled deferreds.  If any fail, change the
        # result to EXCEPTION and log.
        if unhandled:
            unhandled_results = yield defer.DeferredList(unhandled,
                                                         consumeErrors=True)

what happens here is that there are unhandled Deferred needing to be taken care of, but one of them triggers the call to _catchup() which leads to the traceback because self.step._start_unhandled_deferred is now None.

I'm not really aware of the changes that occured since 0.9.0b8 and could explain this, but at this point, isn't it simpler to rewrite MasterShellCommand as a new-style buildstep ? If I understand correctly, the run() base method is the compatibility layer.

comment:3 Changed 4 years ago by gracinet

I did a quick bissection, which says that the regression is a consequence of:

[d17efcab985846551684379bd22d5c1648ddb88e] log.py: change acquire/release by lock.run

I'm stopping there today, will try and confirm tomorrow

Note: See TracTickets for help on using tickets.