add mercurial RSS poller change source

[TedMielczarek]

To use a Mercurial repo as a change source, you currently have to install a hook script in the repo. I've written a change source that polls the RSS feed available in the Hg web interface.

[TedMielczarek]

hgpoller.py (unified diff)

[Pike]

r- from me, urllib.urlopen is evil, as it hangs the buildmaster if the server is slow or down. Use twisted.web.client.getPage instead. I have an example patch in #76, though I need to find out why it has test failures. The other comment I have is that you should pick the revision from the RSS and set it in the changeset. I wonder if there's a halfway cheap way to get the files for that revision from hg, any idea?

[Pike]

duh, #61 that is

[TedMielczarek]

Alright, I can replace urllib.urlopen. I'm not sure what you mean about getting the revision from the RSS, as Mercurial only supports changeset IDs (that I know of), see view-source:http://hg.mozilla.org/mozilla-central/?rss-log

[Pike]

As discussed on #developers, the changeset ID, that is, the numbers like d7e93861f3f3 uniquely identify a snapshot of the repository and thus make a good revision to give to the changeset.

Getting the list of files likely requires modifications to hg, which Ted may or may not do at some point in time or another. Or so. Vague enough? ;-)

[Pike]

I was wondering, would we end up with a repository on the master anyway? If so, it might be more powerful to just pull (and update?) and to get the changes from the local rep directly.

[warner]

no progress on this yet, bumping to 0.7.8

[bhearsum]

fix traceback when change_list is empty

[bhearsum]

I'm working on a different version of this, that works in conjunction with the pushlog extension ( http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/index.cgi/file/76f6d3a9078a/pushlog-feed.py). This works much much better with huge pushes/merges. I'm not sure if this will be as useful to others (since it relies on an external extension). I'll post it here when I'm done and let others decide.

[bhearsum]

hgpoller using pushlog

[bhearsum]

Here's the version we're using now. It requires the 'pushlog' hgweb extension that I mentioned in my last comment. I'm not sure how useful this makes it for others, but I wanted to post it nonetheless.

[Pike]

I frown upon urlopen. Can you use twisted's getPage for the actual loading of the feed so that it's asynch? Like we do in bonsaipoller now, http://buildbot.net/trac/browser/buildbot/changes/bonsaipoller.py#L299.

In _parse_changes, I'd do the check for updated first thing, and continue in the loop. Mozilla early-return style, maybe. I'd be tempted to play with a generator function instead of the for loop, too.

In _process_changes, we got the changes in chronological order, didn't we? I see you doing the max over all changes, instead of just the last?

I read that pushlog has the files in the output, we're not reading those in?

[Pike]

Note, none of the attached versions are still in use on our setups.

[dustin]

I'm with Pike re: using getPage. Once that is fixed, and there are docs and tests, this is good to go.

[bhearsum]

I might have time to fix this up in the next month. No promises, though.

[dustin]

Ted, ben, any chance this is still going to happen, or should we close this as wontfix?

[TedMielczarek]

I don't think we're actually using this code anymore, so we can probably wontfix it.