Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#3066 closed defect (fixed)

buildbot.test.integration.test_master.RunMaster.test_master1 become flaky

Reported by: Ben Owned by: dustin
Priority: critical Milestone: 0.9.0
Version: master Keywords:
Cc:

Description

Discussed there: https://github.com/buildbot/buildbot/pull/1385#issuecomment-64304726

Reproduced this morning on a less powerful hardware (work fine on a big machine.)

Another one failed at the same time:

[ERROR]
Traceback (most recent call last):
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
    result = g.send(result)
  File "/home/benoit/buildbot/buildbot/master/buildbot/buildslave/base.py", line 262, in stopService
    yield service.AsyncMultiService.stopService(self)
  File "/home/benoit/buildbot/buildbot/master/buildbot/util/service.py", line 182, in stopService
    service.Service.stopService(self)
  File "/home/benoit/buildbot/buildbot/master/buildbot/monkeypatches/servicechecks.py", line 33, in stopService
    assert self.running, "%r already stopped" % (self,)
exceptions.AssertionError: <BuildSlave u'local1'> already stopped

buildbot.test.integration.test_master.RunMaster.test_master1
buildbot.test.integration.test_try_client.Schedulers.test_jobdir_wait

djmitche said "I'm not terribly surprised it's flaking"

Change History (19)

comment:1 Changed 6 years ago by Ben

Now three tests are failing:

[ERROR]
Traceback (most recent call last):
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
    result = g.send(result)
  File "/home/benoit/buildbot/buildbot/master/buildbot/buildslave/base.py", line 262, in stopService
    yield service.AsyncMultiService.stopService(self)
  File "/home/benoit/buildbot/buildbot/master/buildbot/util/service.py", line 182, in stopService
    service.Service.stopService(self)
  File "/home/benoit/buildbot/buildbot/slave/buildslave/test/__init__.py", line 43, in stopService
    assert self.running
exceptions.AssertionError: 

buildbot.test.integration.test_master.RunMaster.test_master1
buildbot.test.integration.test_master.RunMaster.test_master2
buildbot.test.integration.test_try_client.Schedulers.test_userpass_no_wait

comment:2 Changed 6 years ago by dustin

This is one of those bugs that "leaks" between tests. In the past, when this has started failing, it's been because a Deferred wasn't handled or a stopService method wasn't called or something like that.

Could you reproduce it *reliably* on this underpowered machine? How underpowered was it? If so, I may try to 'git bisect' on such a system to figure out what changed.

comment:3 Changed 6 years ago by Ben

So far so good, It's not always the same test that is failing, but each try (now #3), at least one is failing ...

Last edited 6 years ago by sa2ajj (previous) (diff)

comment:4 Changed 6 years ago by Dustin J. Mitchell <dustin@…>

In fc5adc8a222ff5c049de26d878ab95ca39385c0a:

mark test_masterN tests as flaky; Refs #3066.

comment:5 Changed 6 years ago by Ben

Unfortunately, those are not the only ones to be flaky:

[ERROR]
Traceback (most recent call last):
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
    result = g.send(result)
  File "/home/benoit/buildbot/buildbot/master/buildbot/buildslave/base.py", line 262, in stopService
    yield service.AsyncMultiService.stopService(self)
  File "/home/benoit/buildbot/buildbot/master/buildbot/util/service.py", line 204, in stopService
    service.Service.stopService(self)
  File "/home/benoit/buildbot/buildbot/master/buildbot/monkeypatches/servicechecks.py", line 33, in stopService
    assert self.running, "%r already stopped" % (self,)
exceptions.AssertionError: <BuildSlave u'local1'> already stopped

buildbot.test.integration.test_try_client.Schedulers.test_userpass_list_builders
buildbot.test.integration.test_try_client.Schedulers.test_userpass_no_wait

When running the test under validate.sh, I now even consistently get KeyboardInterrupt ... Is something very wrong with my system ?

............EEEETraceback (most recent call last):
  File "/home/benoit/buildbot/openvas/venv/bin/trial", line 18, in <module>
    run()
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/scripts/trial.py", line 615, in run
    test_result = trialRunner.run(suite)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 729, in run
    return self._runWithoutDecoration(test, self._forceGarbageCollection)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 756, in _runWithoutDecoration
    run()
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 751, in <lambda>
    run = lambda: suite.run(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 219, in run
    TestSuite.run(self, result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asyncrunner.py", line 36, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 178, in run
    super(LoggedSuite, self).run(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asyncrunner.py", line 36, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 119, in __call__
    return self.run(*args, **kwargs)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 943, in run
    _collectWarnings(self._warnings.append, self._runFixturesAndTest, result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 182, in _collectWarnings
    result = f(*args, **kwargs)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 300, in _runFixturesAndTest
    self._wait(d)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 402, in _wait
    raise KeyboardInterrupt()
KeyboardInterrupt............EEEETraceback (most recent call last):
  File "/home/benoit/buildbot/openvas/venv/bin/trial", line 18, in <module>
    run()
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/scripts/trial.py", line 615, in run
    test_result = trialRunner.run(suite)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 729, in run
    return self._runWithoutDecoration(test, self._forceGarbageCollection)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 756, in _runWithoutDecoration
    run()
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 751, in <lambda>
    run = lambda: suite.run(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 219, in run
    TestSuite.run(self, result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asyncrunner.py", line 36, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 178, in run
    super(LoggedSuite, self).run(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asyncrunner.py", line 36, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 151, in run
    test(result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 119, in __call__
    return self.run(*args, **kwargs)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 943, in run
    _collectWarnings(self._warnings.append, self._runFixturesAndTest, result)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 182, in _collectWarnings
    result = f(*args, **kwargs)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 300, in _runFixturesAndTest
    self._wait(d)
  File "/home/benoit/buildbot/openvas/venv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 402, in _wait
    raise KeyboardInterrupt()
KeyboardInterrupt
Last edited 6 years ago by sa2ajj (previous) (diff)

comment:6 Changed 6 years ago by dustin

Is that occurring after disabling the master tests (comment 4)?

The KeyboardInterrupt? is Trial's (odd) way of saying it's gotten confused by things stopping and starting in a weird order.

It's quite possible that we introduced some other buggy test which "infected" the master tests, and they are actually innocent here.

comment:7 Changed 6 years ago by Ben

It is indeed, when not running the disabled ones.

comment:8 Changed 6 years ago by dustin

In trying to reproduce, I came across #3074, but aside from that I can't replicate this issue..

comment:9 Changed 6 years ago by Ben

So I'm trying my luck at bisect. Between the current master (a9fcf31) and d527f19

On the first try I got: 1323f675a21e272fe65476f952d935e2e7b15660 On the second try I got: 31398aebfc6607edd6191d7a087bcd69da50662f

between a9fcf31 (master) and 967aeb6 (bit older): 334bcd2bf1245daf296a872dce6f54e0779e5810

I'm just running the integration.test_master tests.

It looks like I can't reproduce it very well after all ...

Advices welcome !

comment:10 Changed 6 years ago by dustin

On the latest master, I can make this fail under trial -u after, in my latest attempt, 1095 passes. 0.1% reproducibility is going to make this hard to debug!

comment:11 Changed 6 years ago by dustin

Here's some log output. I've hacked up Twisted's Service.startService and Service.stopService to print what's stopping and starting.

2014-11-30 12:28:59-0500 [-] Log opened.
2014-11-30 12:28:59-0500 [-] --> buildbot.test.integration.test_master.RunMaster.test_master1 <--
2014-11-30 12:28:59-0500 [-] Loading configuration from '/A/trialtemp/basdir/master.cfg'
2014-11-30 12:28:59-0500 [-] Setting up database with URL 'sqlite:/state.sqlite'
2014-11-30 12:28:59-0500 [-] setting database journal mode to 'wal'
2014-11-30 12:28:59-0500 [-] using SQLAlchemy-Migrate version 0.7.2
2014-11-30 12:28:59-0500 [-] Initializing empty database
2014-11-30 12:28:59-0500 [-] Starting BuildMaster -- buildbot.version: 0.8.9
2014-11-30 12:28:59-0500 [-] Loading configuration from '/A/trialtemp/basdir/master.cfg'
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.master.BuildMaster instance at 0x7f0ebc64b3b0>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.metrics.MetricLogObserver instance at 0x7f0ebc64b200>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.cache.CacheManager instance at 0x7f0ebc64b170>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.pbmanager.PBManager instance at 0x7f0ebf3ee1b8>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.buildslave.manager.BuildslaveManager instance at 0x7f0ebc77b950>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.buildslave.protocols.pb.Listener instance at 0x7f0ebc5cd7e8>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.changes.manager.ChangeManager instance at 0x7f0ebc7ed5a8>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.botmaster.BotMaster instance at 0x7f0ebf43e8c0>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.buildrequestdistributor.BuildRequestDistributor instance at 0x7f0ebf43ea28>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.schedulers.manager.SchedulerManager instance at 0x7f0ebf43e320>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.users.manager.UserManagerManager instance at 0x7f0ebc56b710>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.db.connector.DBConnector instance at 0x7f0ebc56b098>
2014-11-30 12:28:59-0500 [-] starting %r <twisted.application.internet.TimerService instance at 0x7f0ebc56b128>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.mq.connector.MQConnector instance at 0x7f0ebf44c5f0>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.mq.simple.SimpleMQ instance at 0x7f0ebc57ffc8>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.data.connector.DataConnector instance at 0x7f0ebc6a1fc8>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.www.service.WWWService instance at 0x7f0ebf5bf290>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.process.debug.DebugServices instance at 0x7f0ebc8438c0>
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.status.master.Status instance at 0x7f0ec1d21488>
2014-11-30 12:28:59-0500 [-] adding 1 new changesources, removing 0
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.changes.pb.PBChangeSource object at 0x7f0ebf3c23d0>
2014-11-30 12:28:59-0500 [-] adding 1 new builders, removing 0
2014-11-30 12:28:59-0500 [-] starting %r <Builder 'u'testy'' at 139701265874888>
2014-11-30 12:28:59-0500 [-] starting %r <twisted.application.internet.TimerService instance at 0x7f0ebf47ee18>
2014-11-30 12:28:59-0500 [-] starting %r <twisted.application.internet.TimerService instance at 0x7f0ebf47ef80>
2014-11-30 12:28:59-0500 [-] PBServerFactory starting on 51418
2014-11-30 12:28:59-0500 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x7f0ebc574c68>
2014-11-30 12:28:59-0500 [-] starting %r <pbmanager.Dispatcher for  on tcp:0>
2014-11-30 12:28:59-0500 [-] trying to load status pickle from /A/trialtemp/basdir/testy/builder
2014-11-30 12:28:59-0500 [-] no saved status pickle, creating a new one
2014-11-30 12:28:59-0500 [-] added builder testy with tags []
2014-11-30 12:28:59-0500 [-] adding scheduler 'all'
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.schedulers.basic.AnyBranchScheduler object at 0x7f0ebf3c2450>
2014-11-30 12:28:59-0500 [-] adding scheduler 'force'
2014-11-30 12:28:59-0500 [-] starting %r <buildbot.schedulers.forcesched.ForceScheduler object at 0x7f0ebf3c2550>
2014-11-30 12:28:59-0500 [-] adding 1 new slaves, removing 0
2014-11-30 12:28:59-0500 [-] BuildMaster is running
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.master.BuildMaster instance at 0x7f0ebc64b3b0>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.status.master.Status instance at 0x7f0ec1d21488>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.debug.DebugServices instance at 0x7f0ebc8438c0>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.www.service.WWWService instance at 0x7f0ebf5bf290>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.data.connector.DataConnector instance at 0x7f0ebc6a1fc8>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.mq.connector.MQConnector instance at 0x7f0ebf44c5f0>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.mq.simple.SimpleMQ instance at 0x7f0ebc57ffc8>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.db.connector.DBConnector instance at 0x7f0ebc56b098>
2014-11-30 12:28:59-0500 [-] stopping %r <twisted.application.internet.TimerService instance at 0x7f0ebc56b128>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.users.manager.UserManagerManager instance at 0x7f0ebc56b710>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.schedulers.manager.SchedulerManager instance at 0x7f0ebf43e320>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.botmaster.BotMaster instance at 0x7f0ebf43e8c0>
2014-11-30 12:28:59-0500 [-] stopping %r <Builder 'u'testy'' at 139701265874888>
2014-11-30 12:28:59-0500 [-] stopping %r <twisted.application.internet.TimerService instance at 0x7f0ebf47ef80>
2014-11-30 12:28:59-0500 [-] stopping %r <twisted.application.internet.TimerService instance at 0x7f0ebf47ee18>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.buildrequestdistributor.BuildRequestDistributor instance at 0x7f0ebf43ea28>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.changes.manager.ChangeManager instance at 0x7f0ebc7ed5a8>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.buildslave.manager.BuildslaveManager instance at 0x7f0ebc77b950>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.buildslave.protocols.pb.Listener instance at 0x7f0ebc5cd7e8>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.pbmanager.PBManager instance at 0x7f0ebf3ee1b8>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.cache.CacheManager instance at 0x7f0ebc64b170>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.process.metrics.MetricLogObserver instance at 0x7f0ebc64b200>
2014-11-30 12:28:59-0500 [-] (TCP Port 51418 Closed)
2014-11-30 12:28:59-0500 [-] Stopping factory <twisted.spread.pb.PBServerFactory instance at 0x7f0ebc574c68>
2014-11-30 12:28:59-0500 [-] stopping %r <pbmanager.Dispatcher for  on tcp:0>
2014-11-30 12:28:59-0500 [-] starting %r <BuildSlave u'local1'>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.changes.pb.PBChangeSource object at 0x7f0ebf3c23d0>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.schedulers.forcesched.ForceScheduler object at 0x7f0ebf3c2550>
2014-11-30 12:28:59-0500 [-] stopping %r <buildbot.schedulers.basic.AnyBranchScheduler object at 0x7f0ebf3c2450>
2014-11-30 12:28:59-0500 [-] Unhandled error in Deferred:
2014-11-30 12:28:59-0500 [-] Unhandled Error
▷⋅⋅⋅Traceback (most recent call last):
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 382, in callback
▷⋅⋅⋅    self._startRunCallbacks(result)
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
▷⋅⋅⋅    self._runCallbacks()
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
▷⋅⋅⋅    current.result = callback(current.result, *args, **kw)
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
▷⋅⋅⋅    _inlineCallbacks(r, g, deferred)
▷⋅⋅⋅--- <exception caught here> ---
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/sandbox/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
▷⋅⋅⋅    result = g.send(result)
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/buildslave/base.py", line 262, in stopService
▷⋅⋅⋅    yield service.AsyncMultiService.stopService(self)
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/util/service.py", line 204, in stopService
▷⋅⋅⋅    service.Service.stopService(self)
▷⋅⋅⋅  File "/home/dustin/code/buildbot/t/buildbot/master/buildbot/monkeypatches/servicechecks.py", line 33, in stopService
▷⋅⋅⋅    assert self.running, "%r already stopped" % (self,)
▷⋅⋅⋅exceptions.AssertionError: <BuildSlave u'local1'> already stopped
▷⋅⋅⋅
2014-11-30 12:28:59-0500 [-] BuildMaster is stopped
2014-11-30 12:28:59-0500 [-] Main loop terminated.

The interesting bit there is that <BuildSlave u'local1'> only gets started after everything has begun shutting down. What's even weirder, though, is that the assertion is failing because self.running is 0, but the starting %r <BuildSlave u'local1'> log is written just before setting self.running = 1.

comment:12 Changed 6 years ago by dustin

I see the problem. It has to do with the async startup of services -- the buildmaster isn't completely started yet (the slave hasn't finished starting) when it is stopped. AsyncMultiService? *should* be able to handle that case, though, so there's a bug here somewhere.

comment:13 Changed 6 years ago by dustin

I can trigger this earlier by heavily loading CPU on my system. It looks like the problem is that BuildslaveManager isn't an AsyncMultiService.

comment:15 Changed 6 years ago by dustin

  • Owner set to dustin
  • Status changed from new to assigned

comment:16 Changed 6 years ago by Ben

Let's close this issue and open new ones for other troubles.

comment:17 Changed 6 years ago by dustin

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:18 Changed 6 years ago by Ben

The flaky markers should also be removed !

comment:19 Changed 6 years ago by dustin

Good point - I'll push that shortly.

Note: See TracTickets for help on using tickets.