Mercurial hook - exceptions.ValueError: signal only works in main thread

[jpichon]

I'm having a problem with using Mercurial and buildbot together. This looks similar to the problem described in #397 (which I found searching for my error thanks to  http://ur1.ca/52q6p), unfortunately applying the suggested change in changes/buildbot.py didn't solve it for me.

I've set up buildbot on a few other machines with Mercurial and a very similar config file just fine, the only major difference I can think of is that this is the first time master and the repositories are on a different machine. Master is set up to listen to Mercurial via PBChangeSource, the hook is like this:

[hooks]
changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook

[hgbuildbot]
master = buildbot:9989

When I push for the first time to the repository, I get the following traceback. The repo is set up to work with HTTP. The changeset does go through to the buildbot master. For this traceback's purpose, I'm pushing with the --debug -v flags, and I also changed hook() in changes/buildbot.py to wrap the function into a try/catch.

1 changesets found
list of changesets:
a7cbcacb6a1034ab436fac506e1fb9e78f3dd56b
bundling: 1/1 changesets (100.00%)
bundling: 1/1 manifests (100.00%)
bundling: thirdfile 1/1 files (100.00%)
sending unbundle command
sending 298 bytes
sending: 0 kb
sending: 0 kb
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: /usr/lib/python2.6/dist-packages/zope/__init__.py:3: UserWarning: Module twisted was already imported from /usr/lib/python2.6/dist-packages/twisted/__init__.pyc, but /usr/local/lib/python2.6/dist-packages is being added to sys.path
remote:   import pkg_resources
remote: Traceback (most recent call last):
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 396, in fireEvent
remote:     DeferredList(beforeResults).addCallback(self._continueFiring)
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 224, in addCallback
remote:     callbackKeywords=kw)
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 213, in addCallbacks
remote:     self._runCallbacks()
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 371, in _runCallbacks
remote:     self.result = callback(self.result, *args, **kw)
remote: --- <exception caught here> ---
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 409, in _continueFiring
remote:     callable(*args, **kwargs)
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1165, in _reallyStartRunning
remote:     self._handleSignals()
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1105, in _handleSignals
remote:     signal.signal(signal.SIGINT, self.sigInt)
remote: exceptions.ValueError: signal only works in main thread
remote: rev a7cbcacb6a1034ab436fac506e1fb9e78f3dd56b sent
checking for updated bookmarks
sending listkeys command

The 2nd changeset I push will hang, with no error message.

1 changesets found
list of changesets:
04b9ca66ef8238f2265bcd798106b673df2648aa
bundling: 1/1 changesets (100.00%)
bundling: 1/1 manifests (100.00%)
bundling: thirdfile 1/1 files (100.00%)
sending unbundle command
sending 301 bytes
sending: 0 kb
sending: 0 kb

I have to kill it. It still goes through to the buildbot master, though.

From then on I will always get the following exception, even though the changes do get through to Master.

1 changesets found
list of changesets:
3eb90d2ce2e2b1d472ff8164801a04a386b11d53
bundling: 1/1 changesets (100.00%)
bundling: 1/1 manifests (100.00%)
bundling: thirdfile 1/1 files (100.00%)
sending unbundle command
sending 298 bytes
sending: 0 kb
sending: 0 kb
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: Traceback (most recent call last):
remote:   File "/usr/local/lib/python2.6/dist-packages/buildbot/changes/hgbuildbot.py", line 178, in hook
remote:     reactor.run()
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1169, in run
remote:     self.startRunning(installSignalHandlers=installSignalHandlers)
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1149, in startRunning
remote:     ReactorBase.startRunning(self)
remote:   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 652, in startRunning
remote:     raise error.ReactorAlreadyRunning()
remote: ReactorAlreadyRunning
checking for updated bookmarks
sending listkeys command

The only way to stop it is to restart Mercurial, and it's back to Traceback #1 above. Before I wrapped hook() in a try/catch statement, the output of that last traceback was only this:

1 changesets found
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: error: changegroup.buildbot hook raised an exception:

The master's logs don't seem to provide information relevant to this. In #397 a more specific traceback is requested, if this one isn't enough let me know where I can get what's needed. Thanks!

[hsorby]

I am also experiencing difficulties with this. I have a few differences though. #397 suggestion did work for me and I stopped getting the exception related to the signal handler.

What happens for me, On the first push I get

  pushing to http://localhost:8000
  searching for changes
  remote: adding changesets
  remote: adding manifests
  remote: adding file changes
  remote: added 1 changesets with 1 changes to 1 files
  remote: rev c34a848fd0f45a56f807638acfcc7f6dbe172039 sent

and everything works great. However on the second push I get:

 pushing to http://localhost:8000
 searching for changes
 remote: adding changesets
 remote: adding manifests
 remote: adding file changes
 remote: added 1 changesets with 1 changes to 1 files
 remote: error: changegroup.buildbot hook raised an exception: 

and nothing gets sent to the build master. As far as my limited tracing has gotten me is to the line of code

  reactor.run()

in the hgbuildbot.hook script this returns an exception from which it never recovers and I have to restart the mercurial server. Thanks.

[hsorby]

I appear to have sorted out my problem. I had to do two things.

1. Change reactor.run() line in hgbuildbot.hook to reactor.run(installSignalHandlers=0)
2. Set the [hgbuildbot] section in .hg/hgrc from my repository where hg serve is running to

    [hgbuildbot]
    master = master.is.here:8080
    branch = my_branch
    fork = true
   

After doing this I have continuous building/testing joy.

[jpichon]

Have you taken a look at your process list after doing this? When I started using "fork" I ended up with a lot of defunct processes (children of Mercurial, one for every push) that wouldn't disappear until Mercurial is restarted. I only left them hanging a few hours, so perhaps they do go away after a while.

My current workaround is to mirror the repositories on the buildbot server, which is less than ideal but does the job for now. (I tried installing buildbot master on the same server than Mercurial, but the problem didn't disappear.)

[dustin]

So here's what's happening:

Mercurial is running the hook in a thread. The hook is using Twisted Python, which really, really does not want to run in a thread. Adding installSignalHandlers=0 avoids the call to signal(..), but has other effects -- notably, Twisted does not get SIGCHLD, so it never cleans up its children.

I'm not sure this can be made to work easily. I suspect that, if we want to keep supporting a mercurial hook, it will need to fire off an external script that submits the change to the master, rather than trying to do so within the Mercurial process.

[hsorby]

Replying to jpichon:

Have you taken a look at your process list after doing this? When I started using "fork" I ended up with a lot of defunct processes (children of Mercurial, one for every push) that wouldn't disappear until Mercurial is restarted. I only left them hanging a few hours, so perhaps they do go away after a while.

No I hadn't but I have now thanks for pointing this out. Yes I do get zombie processes lying around. I can live with this for the mean time, I am just running hg serve on my local machine and I have not shifted it onto a proper web server. I probably won't shift until this is fixed.

[hsorby]

Replying to dustin:

I would like to see this change and hope that you continue to support mercurial.

Is this something that could be changed from the mercurial side? Would it be easier to get mercurial not running this script in another thread?

[dustin]

Oh, we'll definitely continue to support *Mercurial* - the question is, will the hook remain supported :)

Yes, it would be easier to get mercurial to not run this script in another thread, or even in the same process space - best to fork and run it as a separate script. Mixing twisted's runtime libraries with mercurial's is really asking for trouble.

[dustin]

The best solution would be to fire off an HTTP request to the Buildbot change hooks.

[dustin]

This hook no longer invokes Buildbot, so this is indirectly fixed.