Version 10 (modified by marcusl, 10 years ago) (diff)

Added idea on slave-side repo caching

If you're interested in working on any of these ideas, or have ideas of your own that you'd like to work on, get in touch! See the links on Development, or email the project maintainer directly - dustin@….

JavaScript Frontend

Currently, the web frontend is rendered synchronously by the Buildmaster, based on data from the database and from build pickles. This has two disadvantages:

  • the entire Buildmaster is blocked while the page is being rendered, which can be a significant time as build pickles are slow to laod
  • once SQLAlchemy is fully implemented, database queries cannot be made synchronously

The preferred solution at this point is to write the web frontend in browser-side JavaScript, using the existing JSON interface to gather the necessary data.

Windows Process Management

The windows tag documents a number of bugs about starting and killing processes on Windows. Buildbot uses Twisted's process-handling code, so these bugs may be fixed by either re-implementing better support directly in Buildbot, or by patching Twisted's process-handling code.

User objects

After using Darcs for a while, I've become a bit irritated by the fact that my "Changes" column is so wide (since Change objects that are generated by include a full email address as the 'author' field). I'm starting to think that we could have a UserManager? object somewhere (created and registered in the config file) which the ChangeSource can use to translate the author field of the incoming messages into a User object. This User object would have a method to return a suitably short name for the person, for use in the Changes column on the Waterfall display.

Later, this User object might also make it easier to send messages to the particular person by walking through all the available status plugins to see which of them has seen the user recently (I'm thinking IM and IRC here, in particular).

The UserManager? might also be a reasonable place to implement logic like "who is the build sheriff right now?", and to track users that are responsible/interested in different files or modules.

The overall idea would be that most internal buildbot interfaces that accept a string would instead take a User object.

Branch objects

With Darcs in particular (but it also applies to pretty much every VC system), branches are a bit of a hack. The biggest problem I'm seeing right now is when a Builder is configured and presented as being for a specific branch (by virtue of being triggered by a Scheduler which only pays attention to a single branch), then when I use the "force build" button on that Builder, I usually forget to fill in the branch name, and thus get a trunk build by mistake.

Also, we need a way to handle both the "name" of the branch (which is short and generally doesn't include slashes) and the logic that dictates how that branch is interpreted by the VC checkout step (in which some string might be appended to a base repository URL).

Maybe a Branch object would help. The SourceStamp could have one of these instead of a .branch string, and then the VC system could ask it how it wants to be incorporated into the repo URL.

Multi-Project Builds

At the moment, each Build has a single SourceStamp. I'm thinking that to accomodate things like running buildbot version A against twisted version B and python version C, it may be useful to give the Build a dictionary of SourceStamps, to track all the various components that might affect the results. When doing the build, there would be a step to set up the python environment (which might need to download and compile all of python, if the purpose of the test is to make sure that python's TRUNK is still usable, but which hopefully could use a previously cached version most of the time), and then one to set up the Twisted environment, and then one to run the unit tests of the buildbot environment.

Editable description/other fields for builds in WebStatus?

Currently all WebStatus? stuff is read-only (except triggering/canceling builds). It would be cool to be able to do some non-readonly things like:

  • Add some description/notes to build at any time (for example notes to QA)
  • Making builds permanent (so this build will be never removed)
  • Some tags?

Slave-side repository caching =

Slaves often checkout/clone the entire repo (when doing full builds). This can take a lot of time and network traffic, both with CVS:es and DVCS:es.

Having the option for a local clone on the slave side, for each repository used, would reduce network traffic to a minimum and increase checkout/update speed.

  • modern DVCS:es use hardlinks, so local clones are speedy and space-efficient.
  • repeated full cloning of DVCS repos can take a lot of time if the history is big.
  • CVCS:es might want to ease the load of the central server similarly. (f.ex. svnsync can be used to update a local "master" repo and the checkouts can then be done locally using file:// protocol)
  • buildbot try combined with DVCS:es might require the transfer of new commits (not just patches). These "temporary commits" are probably unwanted after the try build, and thus a clean clone is often wanted.
  • also, it might be easier to automate merge/rebase tests, just to see if integration is ok (while throwing away the resulting merge commit in the local repo)