SVNPoller transform_path AssertionError when change includes modifications to files inside and outside the svnurl

[mmacvicar]

SVNPoller transform_path AssertionError? when change includes modifications to files inside and outside the svnurl

I had my SVNPoller set up to monitor svnurl='​http://subversion/trunk/Code'. A checkin was made that changed files in /trunk/Code and /trunk/QA, but the change was ignored and the error below was written to twistd.log. I would expect it to be able to handle checkins that have file modifications inside and outside the svnurl.

2008/11/14 18:09 -0800 [-] svnPoller: _process_changes 10523 .. 10524
2008/11/14 18:09 -0800 [-] Adding change revision 10524
2008/11/14 18:09 -0800 [-] SVNPoller failed
2008/11/14 18:09 -0800 [-] _finished : [Failure instance: Traceback: <type 'exceptions.AssertionError'>: filepath 'trunk/QA/Tests/TriTests/StressShadePrim.cpp'
should start with prefix 'trunk/Code'
        /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:304:_startRunCallbacks
        /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:317:_runCallbacks
        /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:281:_continue
        /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:277:unpause
        --- <exception caught here> ---
        /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:317:_runCallbacks
        /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:413:create_changes
        /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:375:_transform_path
        /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:20:_assert
        ]

I can work around this issue by truncating the svnurl to '​http://subversion' and using split_file to generate (branch, filepath) information, but that wasn't very intuitive.

Reproducible Steps:

1. Set your SVNPoller svnurl to a branch (e.g., svnurl='​http://subversion/trunk/Code')
2. submit a change that include files in that branch and files outside of that branch.

[dustin]

Not entirely clear what it should do with such commits, though -- it's not really testing the entire commit, right?

[miller-mohr]

Replying to dustin:

Not entirely clear what it should do with such commits, though -- it's not really testing the entire commit, right?

Hi,

so first of all thanks for buildbot, I like it and it's helpful :-) And now to the actual comments.

I am a working on a software development project for a scientific simulation code. We use buildbot to run a testsuite including compilation of our project and execution of some test runs (sort of basic regression tests). The testsuite is executed on an HPC cluster at a university computing center. Consequently we only want to execute the testsuite, if there are changes to the source code of the project, but not, if a commit only changes e.g. the doxygen configuration file, or some other part that is not directly related to the actual code base. Otherwise, we'd waste precious HPC resources and risk making the computing centre people real angry ;-)

Thus, from my limited user's point of view, I'd simply expect to trigger buildbot, whenever a commit includes those parts SVNPoller is watching. Something wrong with this POV?

Cheers Marcus

[dustin]

In that case, you should be using fileIsImportant, rather than changing the root of the SVN tree.

This definitely shouldn't fail with an AssertionError, and should probably just ignore the out-of-scope files.

[miller-mohr]

Replying to dustin:

In that case, you should be using fileIsImportant, rather than changing the root of the SVN tree.

This definitely shouldn't fail with an AssertionError, and should probably just ignore the out-of-scope files.

Thanks for the quick reply and the hint. I'll forward the info to my 'remote' buildbot admin.

Cheers Marcus

[tom.prince]

Adding a bunch of people to the CC list, since I'm not sure what the expected behaviour here should be.

[flaviojs]

When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn.

Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.

[dustin]

That's a concise summary of the bug, there!

[Tom Prince]

svnpoller: Don't fail when revisions contain changes outside our prefix.

When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn. Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.

Fixes #385.

Changeset: 1732adcc3eb11477d9fe9258fe772e3defaa3b34

[Tom Prince]

svnpoller: Don't fail when revisions contain changes outside our prefix.

When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn. Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.

Fixes #385.

Changeset: 1732adcc3eb11477d9fe9258fe772e3defaa3b34

[tallfurryman]

Hello,

Sorry to "revive" that thread, but we have a puzzling effect on our master.

$ buildbot --version
Buildbot version: 0.8.7p1
Twisted version: 12.3.0

The code repository has a trunk, and some local externals, thus the master is configured with:

project root (checked out): http://svn.local.lan/svn2/platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat
change source 1: project_root,
change source 2: 'http://svn.local.lan/svn2/platform_1/ref_sources',
change source 3: 'http://svn.local.lan/svn2/sources/tools',
change source 4: 'http://svn.local.lan/svn2/platform_1/vendors/ngr/ckernel/V7',
change source 5: 'http://svn.local.lan/svn2/platform_1/vendors/stu/osp',

And a "svn info --xml" at the root of the slave's svn working copy returns:

$ svn info --xml
(...)
<root>http://svn.local.lan/svn2</root>
(...)

Our problem is that when a commit arrives with one element under the project root and another outside, in an external...

Revision: 18152
Author: someone
Date: vendredi 3 janvier 2014 10:27:48
Message:
[Buildbot] test poller with a root modified file and an external file
----
Modified : /platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat/environment.bat
Modified : /sources/tools/integration_tools/branches/mwv2r/buildbot/buildmaster/mwv2r_hd_xxx/bb_MWV2R.py

...the rev 18152 change is not picked and we get two assertion errors in the master log (slave is a Cygwin XP node). Sorry the log is only an excerpt and there's no way to match the SVNPoller with the change sources, so only guessing here.

2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 17651 .. 17651
2014-01-03 10:32:46+0100 [-] SVNPoller: finished polling None
2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 18142 .. 18152
2014-01-03 10:32:46+0100 [-] Adding change revision 18152
2014-01-03 10:32:46+0100 [-] SVNPoller: Error in  while polling
        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/twisted/internet/_baseprocess.py", line 60, in maybeCallProcessEnded
            proto.processEnded(Failure(reason))
          File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 91, in processEnded
            self.deferred.callback(self.s.getvalue())
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 381, in callback
            self._startRunCallbacks(result)
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 345, in create_changes
            where = self._transform_path(path)
          File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 293, in _transform_path
            (path, self._prefix))
        exceptions.AssertionError: filepath 'platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat/environment.bat' should start with prefix 'sources/tools'

2014-01-03 10:32:46+0100 [-] SVNPoller: no changes
2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 15914 .. 15914
2014-01-03 10:32:46+0100 [-] SVNPoller: finished polling None
2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 18151 .. 18152
2014-01-03 10:32:46+0100 [-] Adding change revision 18152
2014-01-03 10:32:46+0100 [-] SVNPoller: Error in  while polling
        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/twisted/internet/_baseprocess.py", line 60, in maybeCallProcessEnded
            proto.processEnded(Failure(reason))
          File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 91, in processEnded
            self.deferred.callback(self.s.getvalue())
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 381, in callback
            self._startRunCallbacks(result)
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 345, in create_changes
            where = self._transform_path(path)
          File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 293, in _transform_path
            (path, self._prefix))
        exceptions.AssertionError: filepath 'sources/tools/integration_tools/branches/mwv2r/buildbot/buildmaster/mwv2r_hd_xxx/bb_MWV2R.py' should start with prefix 'platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat'

Looking at the code (SVNPoller.create_changes and others), I think the transform_path assertion is causing too much trouble. In our case, the commit is overlapping the project boundaries, the first url breaks on the second prefix, the second url breaks on the first prefix: none of the urls can be processed by SVNPoller, so the change is not picked up. Of course, the commit test urls are built for this example, but we have the issue with branch merges/reintegrations all the like.

I'd be willing to patch, but I'm not sure how and where to catch the exception and properly interrupt/bypass the change source algorithm. Any idea/comment?

[tallfurryman]

Sorry, I just found out how to list changeset 1732adcc3eb11477d9fe9258fe772e3defaa3b34...

Thus, I confirm that the change happily solves our problem. I will update our masters to 0.8.8 if possible, or manually patch the package.