#385 closed defect (fixed)
SVNPoller transform_path AssertionError when change includes modifications to files inside and outside the svnurl
Reported by: | mmacvicar | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | 0.8.8 |
Version: | 0.7.9 | Keywords: | svn |
Cc: | dwlocks, J2ck, flaviojs, bdbaddog |
Description
SVNPoller transform_path AssertionError? when change includes modifications to files inside and outside the svnurl
I had my SVNPoller set up to monitor svnurl='http://subversion/trunk/Code'. A checkin was made that changed files in /trunk/Code and /trunk/QA, but the change was ignored and the error below was written to twistd.log. I would expect it to be able to handle checkins that have file modifications inside and outside the svnurl.
2008/11/14 18:09 -0800 [-] svnPoller: _process_changes 10523 .. 10524 2008/11/14 18:09 -0800 [-] Adding change revision 10524 2008/11/14 18:09 -0800 [-] SVNPoller failed 2008/11/14 18:09 -0800 [-] _finished : [Failure instance: Traceback: <type 'exceptions.AssertionError'>: filepath 'trunk/QA/Tests/TriTests/StressShadePrim.cpp' should start with prefix 'trunk/Code' /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:304:_startRunCallbacks /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:317:_runCallbacks /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:281:_continue /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:277:unpause --- <exception caught here> --- /usr/local/lib/python2.5/site-packages/twisted/internet/defer.py:317:_runCallbacks /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:413:create_changes /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:375:_transform_path /usr/local/lib/python2.5/site-packages/buildbot/changes/svnpoller.py:20:_assert ]
I can work around this issue by truncating the svnurl to 'http://subversion' and using split_file to generate (branch, filepath) information, but that wasn't very intuitive.
Reproducible Steps:
- Set your SVNPoller svnurl to a branch (e.g., svnurl='http://subversion/trunk/Code')
- submit a change that include files in that branch and files outside of that branch.
Change History (14)
comment:1 follow-up: ↓ 4 Changed 11 years ago by dustin
- Milestone changed from undecided to 0.7.+
comment:2 Changed 9 years ago by dustin
- Keywords changes removed
comment:3 Changed 9 years ago by dustin
- Keywords svn added; svnpoller removed
comment:4 in reply to: ↑ 1 Changed 7 years ago by miller-mohr
Replying to dustin:
Not entirely clear what it should do with such commits, though -- it's not really testing the entire commit, right?
Hi,
so first of all thanks for buildbot, I like it and it's helpful :-) And now to the actual comments.
I am a working on a software development project for a scientific simulation code. We use buildbot to run a testsuite including compilation of our project and execution of some test runs (sort of basic regression tests). The testsuite is executed on an HPC cluster at a university computing center. Consequently we only want to execute the testsuite, if there are changes to the source code of the project, but not, if a commit only changes e.g. the doxygen configuration file, or some other part that is not directly related to the actual code base. Otherwise, we'd waste precious HPC resources and risk making the computing centre people real angry ;-)
Thus, from my limited user's point of view, I'd simply expect to trigger buildbot, whenever a commit includes those parts SVNPoller is watching. Something wrong with this POV?
Cheers Marcus
comment:5 follow-up: ↓ 6 Changed 7 years ago by dustin
In that case, you should be using fileIsImportant, rather than changing the root of the SVN tree.
This definitely shouldn't fail with an AssertionError, and should probably just ignore the out-of-scope files.
comment:6 in reply to: ↑ 5 Changed 7 years ago by miller-mohr
Replying to dustin:
In that case, you should be using fileIsImportant, rather than changing the root of the SVN tree.
This definitely shouldn't fail with an AssertionError, and should probably just ignore the out-of-scope files.
Thanks for the quick reply and the hint. I'll forward the info to my 'remote' buildbot admin.
Cheers Marcus
comment:7 Changed 7 years ago by tom.prince
- Cc dwlocks J2ck flaviojs bdbaddog added
Adding a bunch of people to the CC list, since I'm not sure what the expected behaviour here should be.
comment:8 Changed 7 years ago by flaviojs
When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn.
Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.
comment:9 Changed 7 years ago by dustin
That's a concise summary of the bug, there!
comment:10 Changed 7 years ago by tom.prince
- Milestone changed from 0.8.+ to 0.8.8
comment:11 Changed 7 years ago by Tom Prince
- Resolution set to fixed
- Status changed from new to closed
svnpoller: Don't fail when revisions contain changes outside our prefix.
When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn. Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.
Fixes #385.
Changeset: 1732adcc3eb11477d9fe9258fe772e3defaa3b34
comment:12 Changed 7 years ago by Tom Prince
svnpoller: Don't fail when revisions contain changes outside our prefix.
When getting the logs, revisions that don't affect the SVNPoller.svnurl path are already being filtered out by svn. Since the poller doesn't even consider them, I would expect it not to consider unrelated paths that come with the logs and just filter them out.
Fixes #385.
Changeset: 1732adcc3eb11477d9fe9258fe772e3defaa3b34
comment:13 Changed 6 years ago by tallfurryman
Hello,
Sorry to "revive" that thread, but we have a puzzling effect on our master.
$ buildbot --version Buildbot version: 0.8.7p1 Twisted version: 12.3.0
The code repository has a trunk, and some local externals, thus the master is configured with:
project root (checked out): http://svn.local.lan/svn2/platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat change source 1: project_root, change source 2: 'http://svn.local.lan/svn2/platform_1/ref_sources', change source 3: 'http://svn.local.lan/svn2/sources/tools', change source 4: 'http://svn.local.lan/svn2/platform_1/vendors/ngr/ckernel/V7', change source 5: 'http://svn.local.lan/svn2/platform_1/vendors/stu/osp',
And a "svn info --xml" at the root of the slave's svn working copy returns:
$ svn info --xml (...) <root>http://svn.local.lan/svn2</root> (...)
Our problem is that when a commit arrives with one element under the project root and another outside, in an external...
Revision: 18152 Author: someone Date: vendredi 3 janvier 2014 10:27:48 Message: [Buildbot] test poller with a root modified file and an external file ---- Modified : /platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat/environment.bat Modified : /sources/tools/integration_tools/branches/mwv2r/buildbot/buildmaster/mwv2r_hd_xxx/bb_MWV2R.py
...the rev 18152 change is not picked and we get two assertion errors in the master log (slave is a Cygwin XP node). Sorry the log is only an excerpt and there's no way to match the SVNPoller with the change sources, so only guessing here.
2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 17651 .. 17651 2014-01-03 10:32:46+0100 [-] SVNPoller: finished polling None 2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 18142 .. 18152 2014-01-03 10:32:46+0100 [-] Adding change revision 18152 2014-01-03 10:32:46+0100 [-] SVNPoller: Error in while polling Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/twisted/internet/_baseprocess.py", line 60, in maybeCallProcessEnded proto.processEnded(Failure(reason)) File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 91, in processEnded self.deferred.callback(self.s.getvalue()) File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 381, in callback self._startRunCallbacks(result) File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 345, in create_changes where = self._transform_path(path) File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 293, in _transform_path (path, self._prefix)) exceptions.AssertionError: filepath 'platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat/environment.bat' should start with prefix 'sources/tools' 2014-01-03 10:32:46+0100 [-] SVNPoller: no changes 2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 15914 .. 15914 2014-01-03 10:32:46+0100 [-] SVNPoller: finished polling None 2014-01-03 10:32:46+0100 [-] SVNPoller: _process_changes 18151 .. 18152 2014-01-03 10:32:46+0100 [-] Adding change revision 18152 2014-01-03 10:32:46+0100 [-] SVNPoller: Error in while polling Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/twisted/internet/_baseprocess.py", line 60, in maybeCallProcessEnded proto.processEnded(Failure(reason)) File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 91, in processEnded self.deferred.callback(self.s.getvalue()) File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 381, in callback self._startRunCallbacks(result) File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 489, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 576, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 345, in create_changes where = self._transform_path(path) File "/usr/lib/python2.7/site-packages/buildbot/changes/svnpoller.py", line 293, in _transform_path (path, self._prefix)) exceptions.AssertionError: filepath 'sources/tools/integration_tools/branches/mwv2r/buildbot/buildmaster/mwv2r_hd_xxx/bb_MWV2R.py' should start with prefix 'platform_1/product/middleware/branches/mw_v2_ref/mwv2r_hdsat'
Looking at the code (SVNPoller.create_changes and others), I think the transform_path assertion is causing too much trouble. In our case, the commit is overlapping the project boundaries, the first url breaks on the second prefix, the second url breaks on the first prefix: none of the urls can be processed by SVNPoller, so the change is not picked up. Of course, the commit test urls are built for this example, but we have the issue with branch merges/reintegrations all the like.
I'd be willing to patch, but I'm not sure how and where to catch the exception and properly interrupt/bypass the change source algorithm. Any idea/comment?
comment:14 Changed 6 years ago by tallfurryman
Sorry, I just found out how to list changeset 1732adcc3eb11477d9fe9258fe772e3defaa3b34...
Thus, I confirm that the change happily solves our problem. I will update our masters to 0.8.8 if possible, or manually patch the package.
Not entirely clear what it should do with such commits, though -- it's not really testing the entire commit, right?