Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#3454 closed defect (fixed)

Tests failing in Docker with processes not being killed

Reported by: dustin Owned by: dustin
Priority: minor Milestone: 0.9.+
Version: master Keywords:
Cc:

Description

[FAIL]
Traceback (most recent call last):
  File "/buildbot/py27-tw1320/build/slave/buildslave/test/unit/test_runprocess.py", line 657, in check_dead
    self.assertDead(self.child_pid)
  File "/buildbot/py27-tw1320/build/slave/buildslave/test/unit/test_runprocess.py", line 531, in assertDead
    self.fail("pid %d still alive after %ds" % (pid, timeout))
twisted.trial.unittest.FailTest: pid 24771 still alive after 5s

buildslave.test.unit.test_runprocess.TestPOSIXKilling.test_pgroup_usePTY

These tests still run fine for me on my development machine.

Change History (13)

comment:1 Changed 4 years ago by dustin

and a simple run from my working directory mounted in a docker image succeeds as well, so something deeper is going on here.

comment:2 Changed 4 years ago by rutsky

These are the failures seen on metabuildbot? I cannot reproduce them either on my development machine.

comment:4 Changed 4 years ago by dustin

[FAIL]
Traceback (most recent call last):
  File "/buildbot/py27-tw1110/build/slave/buildslave/test/unit/test_runprocess.py", line 714, in check_dead
    self.assertDead(self.child_pid)
  File "/buildbot/py27-tw1110/build/slave/buildslave/test/unit/test_runprocess.py", line 533, in assertDead
    self.fail("pid %d still alive after %ds" % (pid, timeout))
twisted.trial.unittest.FailTest: pid 31229 still alive after 5s

buildslave.test.unit.test_runprocess.TestPOSIXKilling.test_double_fork_no_usePTY

logged ps auxwwf output:

2016-02-14 00:20:53+0000 [-] ps auxwwf:
	USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
	root         1  0.0  0.5 140772 23584 ?        Ssl  Feb08   1:12 /usr/bin/python /usr/local/bin/twistd -ny buildbot.tac
	root       451  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root       479  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root       481  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root       519  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root       590  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root       594  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root      1711  0.0  0.0      0     0 ?        Z    Feb08   0:00 [sleep] <defunct>
	root      1723  0.0  0.0      0     0 ?        Z    Feb08   0:00 [sleep] <defunct>
	root      2422  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python2.7] <defunct>
	root      2771  0.0  0.0      0     0 ?        Z    Feb08   0:00 [python] <defunct>
	root      2938  0.0  0.0      0     0 ?        Z    Feb09   0:00 [sleep] <defunct>
	root      3751  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3753  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3755  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3761  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3769  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3771  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      3857  0.0  0.0      0     0 ?        Z    Feb09   0:00 [sleep] <defunct>
	root      5065  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python] <defunct>
	root      5071  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5400  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python] <defunct>
	root      5832  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5868  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5870  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5915  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5986  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      5988  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6307  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6335  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6337  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6375  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6417  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6419  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6847  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6923  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6925  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6930  0.0  0.0      0     0 ?        Z    Feb09   0:00 [sleep] <defunct>
	root      6936  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6949  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      6951  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      7043  0.0  0.0      0     0 ?        Z    Feb09   0:00 [sleep] <defunct>
	root      7077  0.0  0.0      0     0 ?        Z    Feb09   0:00 [sleep] <defunct>
	root      7734  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      8300  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      8626  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python] <defunct>
	root      9148  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9150  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9152  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9154  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9156  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9158  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9748  0.0  0.0      0     0 ?        Z    Feb09   0:00 [python2.7] <defunct>
	root      9980  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root      9994  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root      9996  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root     10020  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root     10054  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root     10056  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root     10144  0.0  0.0      0     0 ?        Z    Feb10   0:00 [sleep] <defunct>
	root     10195  0.0  0.0      0     0 ?        Z    Feb10   0:00 [sleep] <defunct>
	root     10927  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python2.7] <defunct>
	root     11230  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python] <defunct>
	root     11556  0.0  0.0      0     0 ?        Z    Feb10   0:00 [sleep] <defunct>
	root     11575  0.0  0.0      0     0 ?        Z    Feb10   0:00 [sleep] <defunct>
	root     12367  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python] <defunct>
	root     12625  0.0  0.0      0     0 ?        Z    Feb10   0:00 [python] <defunct>
	root     12808  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     12840  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     12842  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     12895  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     12931  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     12933  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13031  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13125  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13127  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13129  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13157  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13159  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13162  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13195  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13208  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13210  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13311  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13313  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13399  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13501  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13503  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13516  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13530  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13532  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     13653  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     13689  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     13732  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     15331  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     15447  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     15486  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     15804  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python] <defunct>
	root     15964  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16056  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16058  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16125  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16194  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16206  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16208  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16296  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16302  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16304  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16411  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16413  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16423  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16468  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16521  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16529  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16531  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16545  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16547  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16552  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16554  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16585  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16625  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16627  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     16763  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     16797  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     16803  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     16831  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     18572  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     18573  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     18598  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     18953  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19082  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19084  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19112  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19174  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19199  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19202  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19294  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19296  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19309  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19311  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19348  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19351  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19354  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19357  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19381  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19422  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19424  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     19638  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python] <defunct>
	root     19652  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     19683  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     19854  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     21334  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     21463  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     21519  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     21910  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python] <defunct>
	root     22135  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22238  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22240  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22309  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22312  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22320  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22322  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22325  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22327  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22335  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22373  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22375  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     22449  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     22508  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     22529  0.0  0.0      0     0 ?        Z    Feb12   0:00 [sleep] <defunct>
	root     23652  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     23672  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python2.7] <defunct>
	root     24083  0.0  0.0      0     0 ?        Z    Feb12   0:00 [python] <defunct>
	root     24460  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24575  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24577  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24636  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24682  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24714  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24716  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24721  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24724  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24734  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24769  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24771  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     24828  0.0  0.0      0     0 ?        Z    Feb13   0:00 [sleep] <defunct>
	root     24917  0.0  0.0      0     0 ?        Z    Feb13   0:00 [sleep] <defunct>
	root     24946  0.0  0.0      0     0 ?        Z    Feb13   0:00 [sleep] <defunct>
	root     24961  0.0  0.0      0     0 ?        Z    Feb13   0:00 [sleep] <defunct>
	root     26248  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     26302  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     26818  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python] <defunct>
	root     27042  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python] <defunct>
	root     27202  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27241  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27243  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27296  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27343  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27345  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27454  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27555  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27559  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27608  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27664  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27680  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27682  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27687  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27689  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27695  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27731  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27733  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27897  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27908  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27912  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27924  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27940  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27942  0.0  0.0      0     0 ?        Z    Feb13   0:00 [python2.7] <defunct>
	root     27998  0.0  0.0      0     0 ?        Z    Feb13   0:00 [sleep] <defunct>
	root     29243  0.0  0.0      0     0 ?        Z    00:00   0:00 [python2.7] <defunct>
	root     29262  0.0  0.0      0     0 ?        Z    00:00   0:00 [python2.7] <defunct>
	root     29572  0.0  0.0      0     0 ?        Z    00:02   0:00 [python] <defunct>
	root     29946  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     29981  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     29983  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     29985  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     29987  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     29989  0.0  0.0      0     0 ?        Z    00:14   0:00 [python2.7] <defunct>
	root     30061  0.0  0.0      0     0 ?        Z    00:15   0:00 [sleep] <defunct>
	root     30663  0.2  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     30787  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     30866  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     30870  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     30968  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     30971  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31010  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31012  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31025  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31027  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31070  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31132  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31134  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>
	root     31180  0.0  0.0   4452  1600 ?        S    00:20   0:00 /bin/sh -c PYTHON='python2.7'; VE='../sandbox-python2.7-twisted-sqlalchemy==0.7.10samigr-0.7.2'; VEPYTHON='../sandbox-python2.7-twisted-sqlalchemy==0.7.10samigr-0.7.2/bin/python'; PKG_URL='http://ftp.buildbot.net/pub/metabuildbot/python-packages/'; PYGET='import urllib, sys; urllib.urlretrieve(sys.argv[1], filename=sys.argv[2])'; NSP_ARG='--no-site-packages'; # first, set up the virtualenv if it hasn't already been done, or if it's # broken (as sometimes happens when a slave's Python is updated) if ! test -f "$VE/bin/pip" || ! "$VE/bin/python" -c 'import math'; then     echo "Setting up virtualenv $VE";     rm -rf "$VE";     test -d "$VE" && { echo "$VE couldn't be removed"; exit 1; };     mkdir -p "$VE" || exit 1;     # get the prerequisites for building a virtualenv with no pypi access (including both tarballs and wheels)     for prereq in virtualenv.py pip-1.5.6.tar.gz setuptools-5.8-py2.py3-none-any.whl pip-1.5.6-py2.py3-none-any.whl; do         [ -f "$VE/$prereq" ] && continue         echo "Fetching $PKG_URL/$prereq"         $PYTHON -c "$PYGET" "$PKG_URL/$prereq" "$VE/$prereq" || exit 1;     done;     echo "Invoking virtualenv.py (this accesses pypi)"     "$PYTHON" "$VE/virtualenv.py" --python="$PYTHON" $NSP_ARG "$VE" || exit 1 else     echo "Virtualenv already exists" fi; echo "Installing http://ftp.buildbot.net/pub/metabuildbot/python-packages/zope.interface-4.1.1.tar.gz"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" http://ftp.buildbot.net/pub/metabuildbot/python-packages/zope.interface-4.1.1.tar.gz || exit 1; echo "Installing twisted"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" twisted || exit 1; echo "Installing sqlalchemy==0.7.10"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" sqlalchemy==0.7.10 || exit 1; echo "Installing sqlalchemy-migrate==0.7.2"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" sqlalchemy-migrate==0.7.2 || exit 1; echo "Installing multiprocessing==2.6.2.1"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" multiprocessing==2.6.2.1 || exit 1; echo "Installing mock==0.8.0"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" mock==0.8.0 || exit 1; echo "Installing --editable=slave"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" --editable=slave || exit 1; echo "Installing moto==0.3.1"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" moto==0.3.1 || exit 1; echo "Installing boto==2.29.1"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" boto==2.29.1 || exit 1; echo "Installing --editable=master"; "$VE/bin/pip" install --no-index --download-cache="$PWD/../.." --find-links="$PKG_URL" --editable=master || exit 1; if ! test -x "$VE/bin/trial"; then     echo "adding $VE/bin/trial";     ln -s `which trial` "$VE/bin/trial"; fi; echo "Checking for simplejson or json"; "$VEPYTHON" -c 'import json' 2>/dev/null || "$VEPYTHON" -c 'import simplejson' ||             "$VE/bin/pip" install --no-index --download-cache="$PWD/.." --find-links="$PKG_URL" simplejson || exit 1; echo "Checking for sqlite3, including pysqlite3 on Python 2.5"; "$VEPYTHON" -c 'import sqlite3, sys; assert sys.version_info >= (2,6)' 2>/dev/null ||             "$VEPYTHON" -c 'import pysqlite2.dbapi2' ||             "$VE/bin/pip" install --no-index --download-cache="$PWD/.." --find-links="$PKG_URL" pysqlite || exit 1
	root     31281 15.0  0.4  58408 16912 ?        R    00:20   0:00  \_ /buildbot/sqlalchemy__0_7_10-sqlalchemy-migrate__0_7_2/sandbox-python2.7-twisted-sqlalchemy==0.7.10samigr-0.7.2/bin/python2.7 ../sandbox-python2.7-twisted-sqlalchemy==0.7.10samigr-0.7.2/bin/pip install --no-index --download-cache=/buildbot/sqlalchemy__0_7_10-sqlalchemy-migrate__0_7_2/build/../.. --find-links=http://ftp.buildbot.net/pub/metabuildbot/python-packages/ mock==0.8.0
	root     31204 11.3  0.6  88840 26760 ?        S    00:20   0:00 /buildbot/py27-tw1110/sandbox-python2.7-Twisted==11.1.0/bin/python2.7 ../../sandbox-python2.7-Twisted==11.1.0/bin/trial --reporter=bwverbose buildslave.test
	root     31282  0.0  0.0  15892  2472 ?        R    00:20   0:00  \_ ps auxwwf
	root     31207 50.6  2.2 450208 91104 ?        Sl   00:20   0:03 /buildbot/py27-tw1400/sandbox-python2.7-Twisted==14.0.0/bin/python2.7 ../../sandbox-python2.7-Twisted==14.0.0/bin/trial --reporter=bwverbose buildbot.test
	root     31229  0.0  0.0      0     0 ?        Z    00:20   0:00 [python2.7] <defunct>

comment:5 Changed 4 years ago by rutsky

It's zombie apocalypse :)

comment:6 Changed 4 years ago by dustin

So it looks like one of the problems is that the twistd process is not acting like init and reaping re-parented children. Which wouldn't be so bad if that twistd process didn't hang around forever.

In trying to investigate, a few runs of 'docker ps' and 'docker exec' failed in odd ways, and blammo, the whole container is gone.

So either this was a fluke that "poisoned" the container, or the version of Docker on this VM is just nonfunctional (but it's 1.9.1 which isn't so old). I'll restart it now.

comment:9 Changed 4 years ago by dustin

https://github.com/Yelp/dumb-init is probably a good fix, and avaialable as a static binary.

comment:10 Changed 4 years ago by tardyp

I think dumb-init is fine, but that would probably be uber cool to write a small reaper for twisted that we could embed in our workers. as really a worker is meant to spawn childs.

I think reaping is really just about registering to SIG_CHILD event and calling waitpid()

comment:11 Changed 4 years ago by dustin

https://github.com/buildbot/metabbotcfg/pull/34

Twisted does reap its own children -- it just doesn't reap zombie grandchildren, which is really something only the init process needs to be concerned about.

comment:12 Changed 4 years ago by dustin

  • Resolution set to fixed
  • Status changed from new to closed

The linux worker is updated. I pinged Bill to ask him to update. From a metabuildbot run, this seems to have helped! Yay!

comment:13 Changed 4 years ago by dustin

  • Summary changed from Tests failing with newer Twisteds in Docker configuration to Tests failing in Docker with processes not being killed
Note: See TracTickets for help on using tickets.