SVN - allow multiple repositories

[ipv6guru]

Our project is divided into various components, some of which are spread over multiple different subversion repositories. Many of these components eventually assemble into a single project, so having multiple buildmasters didn't really make much sense for us. We wanted to be able to trigger a different set of builders depending on in which repository a commit was made.

This patch allows a scheduler to specify a "repository" parameter, in much the same way as the existing branch parameter. If the parameter is not "None" the scheduler will filter out changes that are not sent with a matching repository.

The subversion change sender is modified to transmit the repository value across to the ChangeSource and the Change itself has a "repository" field added to it.

Comment 1:

An example from our master.cfg of a scheduler that only wants to wake up when a change is from a certain repository:

cschedulers?.append(Scheduler(name="foundationmunged-sched", repository="/data/svn/foundationmunged", branch=None, treeStableTimer=1*5, builderNames=["foundationmunged-linux", "foundationmunged-mac"]))

Note that "repository" is a required argument and must be specified as "None" if you want to wake up on a change to any (or unspecified) repositories. This is done to be consistent with the way that "branch" is being handled; it would be easy to make the repository field optional if that is preferable.

[ipv6guru]

Moved from sourceforge

[mue]

I will try to solve this issue, but first I want to make some things clear. I'm not sure if the patches are the right approach. We have the problem of multiple repositories and one buildmaster, too. We solved it with a new class RepositoryScheduler? which inherits from AnyBranchScheduler?:

class RepositoryScheduler(AnyBranchScheduler):
    """Extend Scheduler to allow multiple projects"""
    def __init__(self, name, branches, treeStableTimer, builderNames,
                 repository, fileIsImportant=None, properties={}):
        """ Override Scheduler.__init__ Add a new parameter : repository """
        AnyBranchScheduler.__init__(self, name, branches, treeStableTimer,
                                    builderNames, fileIsImportant, properties)
        self.repository = repository

    def addChange(self, change):
        """Call Scheduler.addChange only if the repository is modified"""
        # check to make sure the repository matches!
        repo = change.repository
        #log.msg("DEBUG: %s -> %s" % (str(repo), str(change.comments)))
        if repo is not None:
            log.msg('checking %s vs %s' % (repo, self.repository))
            if repo != self.repository:
                log.msg("%s ignoring repository %s" % (self, repo))
                return
        # call our parent since this on the correct repository
        AnyBranchScheduler.addChange(self, change)

It is used together with SVNPoller, the master.cfg looks like this:

svn_source = SVNPoller("https://our.company/svn/Project1/Module1",
                       svnuser=const_svnuser, svnpasswd=const_svnpasswd,
                       split_file=my_split_file,
                       pollinterval=const_pollinterval)
c['change_source'].append(svn_source)

svn_source = SVNPoller("https://our.company/svn/Project2/Module2",
                       svnuser=const_svnuser, svnpasswd=const_svnpasswd,
                       split_file=my_split_file,
                       pollinterval=const_pollinterval)
c['change_source'].append(svn_source)

sch_Module1 = RepositoryScheduler(name="Module1",
                                  branches=None,
                                  repository="https://our.company/svn/Project1/Module1",
                                  treeStableTimer=defaultTreeStableTime,
                                  builderNames=["Module1"])
c['schedulers'].append(sch_Module1)

sch_Module2 = RepositoryScheduler(name="Module2",
                                  branches=None,
                                  repository="https://our.company/svn/Project2/Module2",
                                  treeStableTimer=defaultTreeStableTime,
                                  builderNames=["Module2"])
c['schedulers'].append(sch_Module2)

I'm not sure if we want the repository-feature in the default scheduler.

[dustin]

This sounds like a good idea. I wasn't aware that changes have a "repository" attribute, but this sounds perfect. Would it be more natural to merge this idea of selecting only changes from certain repositories back into the existing scheduler classes, instead of creating a new class?

Also, some documentation of how to use this would be great.

This will be a *huge* addition to the next release -- people ask for this feature *daily*!

[mue]

Well, I just looked into my class Change and I must admit, that the "repository" attribute is an addition of me. I have overlooked this, sorry. But maybe this should be part of the solution:

--- buildbot-0.7.10p1/buildbot/changes/changes.py       2009-03-02 18:13:02.000000000 +0100
+++ b/buildbot/changes/changes.py      2009-03-02 19:34:47.000000000 +0100
@@ -54,7 +54,7 @@
     revision = None # used to create a source-stamp

     def __init__(self, who, files, comments, isdir=0, links=[],
-                 revision=None, when=None, branch=None, category=None):
+                 revision=None, when=None, branch=None, category=None, repository=None):
         self.who = who
         self.comments = comments
         self.isdir = isdir
@@ -65,6 +65,7 @@
         self.when = when
         self.branch = branch
         self.category = category
+        self.repository = repository

         # keep a sorted list of the files, for easier display
         self.files = files[:]
diff -ru buildbot-0.7.10p1/buildbot/changes/svnpoller.py buildbot-0.7.10-tt/buildbot/changes/svnpoller.py
--- buildbot-0.7.10p1/buildbot/changes/svnpoller.py     2009-03-02 18:13:02.000000000 +0100
+++ b/buildbot/changes/svnpoller.py    2009-03-02 19:34:47.000000000 +0100
@@ -439,7 +439,8 @@
                                files=files,
                                comments=comments,
                                revision=revision,
-                               branch=branch)
+                               branch=branch,
+                               repository=self.svnurl)
                     changes.append(c)

         return changes

I will start working on this, some work is left in my current solution.

[mue]

Replying to dustin:

Would it be more natural to merge this idea of selecting only changes from certain repositories back into the existing scheduler classes, instead of creating a new class?

I think we should merge this feature back into the existing scheduler classes. Would it be better to make the parameter "repository" optional or not? I think required like "branch" is better, but this will break every existing master.cfg.

[dustin]

Yeah, it has to remain optional. But let's not give None a special significance like has happened for branches - that has been endlessly confusing. Rather, if a Change has a repository of None, then a scheduler which is filtering only changes from a particular repository should log a warning and ignore it. Then existing master.cfg's (with no repository in the Change objects and no repository-filtering going on in schedulers) will continue to work, and users applying this new technique will be required to annotate all Change objects with a repository.

[dustin]

Can this be solved with Change categories? Essentially by inventing a category per repository?

[jcoffland]

I think this is a great feature, but I agree it would be better served by a 'category' parameter.

• Allow the user to label each ChangeSource with a 'category' tag.
• Add any 'category' tags to Changes produced by those ChangeSources?.

Change already has a 'category' parameter and Scheduler can already filter by Change 'category'.

This would make for a more flexible feature which could even replace the need for branch based filtering.

[jcoffland]

A very simple patch which adds the 'category' feature to SVNPoller

[mue]

Well, i agree this could be solved with the "category". But I already implemented it with "repository" (see my github-branch). At the moment I'm testing this. Sorry for not being active here for the last weeks, but I was busy.

[duncs]

From my POV I would prefer repository to be dealt with separately - I already use category for something else.

I have a number of projects all built out of one buildbot master - I use categories so I can have different waterfalls for each product and an overall waterfall if I want to see everything. Some of those product streams can use 2 repositories to build the whole product.

However, I cannot use Scheduler since a change in one repository tries to kick off builds across all repositories. For this reason I could do with a RepositoryScheduler? to limit builds to the correct repos. Category by itself is not enough for me, as far as I can see, due to having multiple repos to one category (aka product stream)

[dustin]

mue and duncs make a good point that category is really not the appropriate way to handle this. mue -- which branch? should I pull it, or is it still in progress?

[mue]

Well, I'm still very busy at work. I think I've done the main coding, but the tests are still missing. I would prefer to make some more testing before I release my branch.

[Nicolas]

Isn't this already supported through categories?

[dustin]

mue - any update

[mue]

I have a working version ready, on github in my bug467-branch. This branch is based on an older master-branch, so my changes are maybe not mergeable. I'm not familiar enough with git to solve this, maybe I can get some help? I played a little bit with git rebase, but I got not the expected result.

[marcusl]

You can always merge the master branch to your branch, and resolve any conflicts. That would make it up to date. (If there are too many conflicts, some intermediate merges might help, if you can identify some good spots in the history to merge with.)

In order to make the history look good, one would need to rebase, but I find that hard when there's many conflicts (however, I'm no git expert...).

After merge, one could export the diff to master as a patch and then create a new commit with that patch, then ignore/scrap the merge and old commits.

[mue]

Thanks for the help.

I merged the master branch into my bug branch and now it is up to date. But now my tests are failing. I have to take a closer look at the changes in the master branch.

[mue]

I just noticed, that merging the master branch was a bad idea. I should have used the buildbot-0.7.12rc1 tag. I will try it again at the weekend.

[marcusl]

Oh. I thought you were aiming for 0.8. For 0.7.12rcX, the 'release' branch is better as it contains some fixes on top of buildbot-0.7.12rc1, scheduled to be buildbot-0.7.12rc2 (and eventually 0.7.12 proper).

[mue]

I wasn't sure about the right release target. At the moment the situation is like this: it is too late for 0.7.12, and for 0.8 I can't do it. I tried to merge it with the current master branch, which is for 0.8. But in parallel the work for the db-support is going on, which is changing too much in the files I have to touch. I don't have the time to test my implementation with these changes, because I think I have to setup an new environment for this (at the moment many tests are failing which worked before the merge). Maybe I should have taken some time for this ticket earlier.

[marcusl]

For 0.8, all the db work is done in Brian's (aka. warner) scheduler-db-rebases branch, so none of that should be merged to the master branch.

However, if you manage to merge cleanly with 'release' your changes will be ok on top of 0.7.12, and we can pull that later and merge it to master/0.8 after 0.7.12 is out. (That's a good middle-ground if you aren't able to merge up to with master.)

[chrb]

I was surprised to see that buildbot does not already support multiple repositories. I have attached buildbot-0.7.12-multiple-repository-support.diff which is Mue's patches plus some updates to the debug client to allow repository to be specified in a fake change (you will also need the fix for bug #719 to run the debug client as it is broken in 0.7.12). The change summary will also print the repository. I am not really sure from reading the above comments why Mue's patches were not merged earlier - it only adds about 25 lines of code. Is there some other concern?

[chrb]

btw I am not using the requestBuild function of debug client - I force build requests through the web interface instead. The use case and functionality of specifying a particular repository and forcing a build for it from the debug client probably does not work.

[chrb]

Replying to chrb:

btw I am not using the requestBuild function of debug client - I force build requests through the web interface instead. The use case and functionality of specifying a particular repository and forcing a build for it from the debug client probably does not work.

It works to specify a particular builder. However you can not override the SVN url for that builder. I doubt the functionality is that useful for most people and I have no need for it, but maybe someone will add this in future if required.

[chrb]

buildbot-0.7.12-multiple-repository-support.diff

[dustin]

A similar, but more advanced version, of this has been merged into 0.8.0. Check it out, and file bugs for anything that we missed?

[chrb]

I tried 0.8.0 and filed bug #754 and bug #755. It looks like 0.8.0 adds the repository attribute to a change? One thing that I had problems with in the previous solution was that the repository field was not updated by the contrib/git_buildbot.py script. Does 0.8.0 support the repository field with git repositories? Also has the manual been updated - it would be useful to have an example use case - presumably the idea is to create a scheduler class like class RepositoryScheduler from the above code?

[dustin]

Well, the manual doesn't have use cases, but it is documented. The Scheduler class can now filter on all sorts of things, including repository and project.

It doesn't look like git_buildbot.py got updated, but that should be easy to patch -- send it along!

[Dustin J. Mitchell]

Add repository support to svn_buildbot.py

Fixes #924, refs #467

Changeset: ead23e5f4b7d3a903835cd684d448bef6b6359e0