Opened 5 years ago

Closed 5 years ago

#2970 closed defect (fixed)

latent slaves fail

Reported by: dustin Owned by: Mikhail Sobolev <mss@…>
Priority: critical Milestone: 0.9.0
Version: master Keywords: latent
Cc:

Description

2014-10-25 22:45:21+0200 [Broker,1,192.168.178.29] Slave docker received connection while not trying to substantiate.  Disconnecting.
2014-10-25 22:45:21+0200 [Broker,1,192.168.178.29] Peer will receive following PB traceback:
2014-10-25 22:45:21+0200 [Broker,1,192.168.178.29] Unhandled Error
	Traceback (most recent call last):
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbot/master/buildbot/pbmanager.py", line 170, in call_attached
	    d = defer.maybeDeferred(persp.attached, mind)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
	    result = f(*args, **kw)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1237, in unwindGenerator
	    return _inlineCallbacks(None, gen, Deferred())
	--- <exception caught here> ---
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
	    result = g.send(result)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/protocols/pb.py", line 97, in attached
	    yield self.buildslave.attached(self)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 719, in attached
	    self._disconnect(bot)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 434, in _disconnect
	    tport = slave.broker.transport
	exceptions.AttributeError: 'Connection' object has no attribute 'broker'
	
--------------
and:

2014-10-25 22:32:48+0200 [Broker,0,192.168.178.29] slave 'docker' attaching from IPv4Address(TCP, '192.168.178.29', 51831)
2014-10-25 22:32:48+0200 [Broker,0,192.168.178.29] Got slaveinfo from 'docker'
2014-10-25 22:32:48+0200 [-] bot attached
2014-10-25 22:32:48+0200 [Broker,0,192.168.178.29] Peer will receive following PB traceback:
2014-10-25 22:32:48+0200 [Broker,0,192.168.178.29] Unhandled Error
	Traceback (most recent call last):
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback
	    self._startRunCallbacks(fail)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
	    self._runCallbacks()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
	    _inlineCallbacks(r, g, deferred)
	--- <exception caught here> ---
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/protocols/pb.py", line 97, in attached
	    yield self.buildslave.attached(self)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 374, in attached
	    yield self.updateSlave()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 884, in _sent
	    d1 = sb.attached(self, remote, self.slave_commands)
	exceptions.TypeError: attached() takes exactly 3 arguments (4 given)
	
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] slave 'docker' attaching from IPv4Address(TCP, '192.168.178.29', 51834)
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Got duplication connection from 'docker' starting arbitration procedure
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Got error while trying to ping connected slave docker:Calling Stale Broker
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Old connection for 'docker' was lost, accepting new
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Got slaveinfo from 'docker'
2014-10-25 22:33:24+0200 [-] bot attached
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Peer will receive following PB traceback:
2014-10-25 22:33:24+0200 [Broker,1,192.168.178.29] Unhandled Error
	Traceback (most recent call last):
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback
	    self._startRunCallbacks(fail)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
	    self._runCallbacks()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
	    _inlineCallbacks(r, g, deferred)
	--- <exception caught here> ---
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/protocols/pb.py", line 97, in attached
	    yield self.buildslave.attached(self)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 374, in attached
	    yield self.updateSlave()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 884, in _sent
	    d1 = sb.attached(self, remote, self.slave_commands)
	exceptions.TypeError: attached() takes exactly 3 arguments (4 given)
	
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] slave 'docker' attaching from IPv4Address(TCP, '192.168.178.29', 51836)
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Got duplication connection from 'docker' starting arbitration procedure
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Got error while trying to ping connected slave docker:Calling Stale Broker
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Old connection for 'docker' was lost, accepting new
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Got slaveinfo from 'docker'
2014-10-25 22:33:42+0200 [-] bot attached
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Peer will receive following PB traceback:
2014-10-25 22:33:42+0200 [Broker,2,192.168.178.29] Unhandled Error
	Traceback (most recent call last):
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 423, in errback
	    self._startRunCallbacks(fail)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
	    self._runCallbacks()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
	    _inlineCallbacks(r, g, deferred)
	--- <exception caught here> ---
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/protocols/pb.py", line 97, in attached
	    yield self.buildslave.attached(self)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 374, in attached
	    yield self.updateSlave()
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 884, in _sent
	    d1 = sb.attached(self, remote, self.slave_commands)
	exceptions.TypeError: attached() takes exactly 3 arguments (4 given)

Change History (7)

comment:1 Changed 5 years ago by dustin

I'm guessing this is related to the slave-protocol work??

comment:2 Changed 5 years ago by Ben

I strongly believe it is, as the very first error I had when investigating this area was:

2014-10-25 18:52:24+0200 [-] starting build <Build manager> using slave <LatentSlaveBuilder builder=u'manager' slave=u'docker'>
2014-10-25 18:52:24+0200 [-] acquireLocks(slave <DockerLatentBuildSlave u'docker'>, locks [])
2014-10-25 18:52:24+0200 [-] substantiating slave <LatentSlaveBuilder builder=u'manager' slave=u'docker'>
2014-10-25 18:52:24+0200 [-] while preparing slavebuilder:
	Traceback (most recent call last):
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
	    result = g.send(result)
	  File "/Users/benoit/Python/buildbot/master/buildbot/process/builder.py", line 511, in maybeStartBuild
	    build_started = yield self._startBuildFor(slavebuilder, breqs)
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1237, in unwindGenerator
	    return _inlineCallbacks(None, gen, Deferred())
	  File "/Users/benoit/Python/buildbotEnv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
	    result = g.send(result)
	--- <exception caught here> ---
	  File "/Users/benoit/Python/buildbot/master/buildbot/process/builder.py", line 345, in _startBuildFor
	    ready = yield slavebuilder.prepare(self.builder_status, build)
	  File "/Users/benoit/Python/buildbot/master/buildbot/process/slavebuilder.py", line 226, in prepare
	    d = self.substantiate(build)
	  File "/Users/benoit/Python/buildbot/master/buildbot/process/slavebuilder.py", line 246, in substantiate
	    d = self.slave.substantiate(self, build)
	  File "/Users/benoit/Python/buildbot/master/buildbot/buildslave/base.py", line 679, in substantiate
	    if self.slave is None:
	exceptions.AttributeError: 'DockerLatentBuildSlave' object has no attribute 'slave'
	
2014-10-25 18:52:24+0200 [-] slave <Build manager> can't build <LatentSlaveBuilder builder=u'manager' slave=u'docker'> after all; re-queueing the request
2014-10-25 18:52:24+0200 [-] releaseLocks(<DockerLatentBuildSlave u'docker'>): []

Where the Latentslave object is suposed to have a slave property of type buildbot.buildslave.protocols.base.Connection (According to the call of notifyOnDisconnect here: https://github.com/buildbot/buildbot/blob/master/master/buildbot/buildslave/base.py#L432)

comment:3 Changed 5 years ago by Ben

How surprising, test coverage for the process/slavebuilder.py and buildslaves/base.py is very low ! 11th and 3rd when sorting by amount of missed lines ...

comment:4 Changed 5 years ago by sa2ajj

What Docker latent slave is being used here?

comment:5 Changed 5 years ago by Ben

It's my one ;) , based on the one from dockerbot. As soon as I'm proficient enough with git branches, I'll try to make a development branch on my side with it. unfortunately, development there is a bit stalled due to these particular issueS ... It's not fun to debug your own class when the world around you is collapsing in all directions ...

comment:6 Changed 5 years ago by sa2ajj

I filed a ticket -- #2956 -- and it would be great to have one :)

Should you need help with git branches, do not hesitate to ask.

comment:7 Changed 5 years ago by Mikhail Sobolev <mss@…>

  • Owner set to Mikhail Sobolev <mss@…>
  • Resolution set to fixed
  • Status changed from new to closed

In ce5389e7a2ae863750b0f980ca83a641fef02d7b:

Merge pull request #1341 from benallard/fix-abstractlatentbuildslave

Fix AbstractLatentBuildslave?

Fixes ticket:2970

Note: See TracTickets for help on using tickets.