Changes between Version 12 and Version 13 of ProjectIdeas

Mar 27, 2011, 5:00:37 AM (10 years ago)



  • ProjectIdeas

    v12 v13  
    11If you're interested in working on any of these ideas, or have ideas of your own that you'd like to work on, get in touch!  See the links on [wiki:Development Development], or email the project maintainer directly -
     3== Your Idea Here! ==
     4We are open to any ideas that will make Buildbot better - think of this page as a set of ideas to get you started.
    36== !JavaScript Frontend ==
    57Currently, the web frontend is rendered synchronously by the Buildmaster, based on data from the database and from build pickles.  This has two disadvantages:
    68 * the entire Buildmaster is blocked while the page is being rendered, which can be a significant time as build pickles are slow to laod
    7  * once SQLAlchemy is fully implemented, database queries cannot be made synchronously
     9 * once SQLAlchemy is fully implemented (which will happen before GSoC 2011 begins), database queries cannot be made synchronously
    810The preferred solution at this point is to write the web frontend in browser-side !JavaScript, using the existing JSON interface to gather the necessary data.
     12=== Scope ===
     13Work on this project would begin with a simple proof-of-concept page, selection of a !JavaScript framework, and the necessary Python code to support all of it.  This will involve a lot of discussion about problems that may come up later in the project, and planning ahead for them.  Once this has been reviewed and accepted, you can go page-by-page through the Buildbot interface, rewriting each (or even parts of each) in !JavaScript.
     15The JS frontend will talk to a REST service provided by the Buildmaster.  Much of this service is already implemented, but it will need to be expanded and improved, which will require some Python programming and learning a bit about Twisted Python
     17Issues to consider:
     18 * testing - how will we test the JS?
     19 * compatibility - what browsers will we support?
     20 * configuration - how can we configure the frontend using parameters in the Buildmaster's configuration file?
     21 * security - how can we protect information from being seen by those who should not see it?  How can we limit who can start or stop builds?
     22 * licensing - if we redistribute a JS framework, its license must be compatible with Buildbot's
     23 * documentation - how can we do a better job of documenting the web frontend for others to use?
    1025== Windows Process Management ==
    1227The [tag:windows windows] tag documents a number of bugs about starting and killing processes on Windows.  Buildbot uses Twisted's process-handling code, so these bugs may be fixed by either re-implementing better support directly in Buildbot, or by patching Twisted's process-handling code.
    14 == User objects ==
     29See also:
     30 *
    16 After using Darcs for a while, I've become a bit irritated by the fact that my "Changes" column is so wide (since Change objects that are generated by include a full email address as the 'author' field). I'm starting to think that we could have a UserManager object somewhere (created and registered in the config file) which the ChangeSource can use to translate the author field of the incoming messages into a User object. This User object would have a method to return a suitably short name for the person, for use in the Changes column on the Waterfall display.
     32=== Scope ===
     33To do well with this project, you would need to bring a lot of the Windows experience that is lacking in the Buildbot development community.  Assuming you're familiar with Windows APIs and accessing them from Python, this bug would entail
     34 * writing test cases to reproduce bugs users have seen
     35 * interacting with the Twisted community to design solutions that can be merged upstream
     36 * implementing portable fixes to those bugs
     37 * documenting them
    18 Later, this User object might also make it easier to send messages to the particular person by walking through all the available status plugins to see which of them has seen the user recently (I'm thinking IM and IRC here, in particular).
     39== User Objects ==
     40Buildbot deals with "users" in a number of ways: commits are generated by users, and those cause builds that the users may want to know about.  Users can also cause builds via the web, IRC, or from command-line tools.  Buildbot communicates results back to people through a number of mechanisms, too: email, IRC, web, and so on.  In many cases, Buildbot administrators need to ensure that only certain people can perform certain operations, such as starting or cancelling builds.
    20 The UserManager might also be a reasonable place to implement logic like "who is the build sheriff right now?", and to track users that are responsible/interested in different files or modules.
     42The project, then, is to design and implement a consistent way to represent "users" throughout Buildbot.
    22 The overall idea would be that most internal buildbot interfaces that accept a string would instead take a User object.
     44=== Scope ===
     45This project has a significant design component, since there is very little code related to users in Buildbot right now.  The design would need to address backward compatibility (users should not have to change their buildbot configurations when upgrading), security, flexibility (we have almost a dozen version control backends, and each thinks about users differently!), and configurability.  There will most likely be a new database table or two to add.
    24 == Branch objects ==
     47Once the design is done, this project should have a few concrete milestones where particular parts of Buildbot start using the new functionality.  For example, when MailNotifier can use the new functionality to translate the author of a source-code commit into an email address, that is a concrete milestone.
    26 With Darcs in particular (but it also applies to pretty much every VC system), branches are a bit of a hack. The biggest problem I'm seeing right now is when a Builder is configured and presented as being for a specific branch (by virtue of being triggered by a Scheduler which only pays attention to a single branch), then when I use the "force build" button on that Builder, I usually forget to fill in the branch name, and thus get a trunk build by mistake.
     49== Multi-Repository Builds and Build Coordination ==
     50As implemented right now, a single Buildmaster can build multiple projects, but each project must be built from a single source code repository, and builds of the projects must not interact.  Increasingly, though, users want to build projects that require source from multiple repositories, and where a checkin to any of those repositories should trigger a build.  Similarly, users often want to coordinate multiple projects, for example a web frontend and backend server that are built separately but must be tested together or packaged into a single installer.
    28 Also, we need a way to handle both the "name" of the branch (which is short and generally doesn't include slashes) and the logic that dictates how that branch is interpreted by the VC checkout step (in which some string might be appended to a base repository URL).
     52See also:
     53 *
     54 *
    30 Maybe a Branch object would help. The !SourceStamp could have one of these instead of a .branch string, and then the VC system could ask it how it wants to be incorporated into the repo URL.
     56=== Projects and Scope ===
     57Fixing this is a big, ongoing task that cannot be completed in a summer, so this project would involve *improving* support for these sorts of configurations.  Part of the project (and proposal), then, is defining what problem you'd like to solve.  We have discussed a few ideas that might get you started:
    32 == Multi-Project Builds ==
     59 * Sourcestamps that point to multiple repositories and revisions in those repositories.  Builds would then specify a way to check out the given revision of each repository before beginning the compile and test steps.  Designing these would involve some significant changes to the database schema, as well as a solid plan for backward compatibility.  Because changes would be required on the master and slave, you would need to think about compatibility between different versions of master and slaves, as well.  If possible, it would be smart to concentrate on only one version-control system for a summer project, so that others could fill in the blanks later.
     60 * Source managers that know everything about a particular version control repository.  A source manager, for example, could determine the changes that were made between two source stamps (revisions), or provide the information required to check out a particular revision from a repository.  Source managers, too, could be implemented for a single version control system, so that others can finish the work for the remaining systems.
     61 * "Gaggles" of builds.  Right now, it is very difficult to wait for a number of different builds to finish - for example, if you need the Windows and Linux builds to finish so that you can compress them into a CD image.  The most difficult part is to determine when two builds "go together", even if they occur at different times.  If we can group builds into gaggles, then it's easy to wait for, for example, all of the Linux and Windows builds in a particular gaggle to finish before starting the CD-burning build.  This project would involve a lot of design discussions with Buildbot users, then some database and code changes to manage gaggle identifiers.  Finally, some easy-to-use synchronization steps and documentation of how to use them would make this available to Buildbot's users.
    34 At the moment, each Build has a single !SourceStamp. I'm thinking that to accomodate things like running buildbot version A against twisted version B and python version C, it may be useful to give the Build a dictionary of !SourceStamps, to track all the various components that might affect the results. When doing the build, there would be a step to set up the python environment (which might need to download and compile all of python, if the purpose of the test is to make sure that python's TRUNK is still usable, but which hopefully could use a previously cached version most of the time), and then one to set up the Twisted environment, and then one to run the unit tests of the buildbot environment.
     63== Consistent, Efficient Source Steps ==
    36 == Editable description/other fields for builds in !WebStatus ==
     65Slaves often checkout/clone the entire repo (when doing full builds). This can take a lot of time and network traffic, and often takes *more* time with the new distributed version-control systems, since they like to download the entire history for each clone.  There are a few "modes" defined which try to reduce this need (e.g., by copying an untouched checkout, or trying to "clean" the build artifacts out of a checkout), but these are implemented inconsistently across version control systems.  They also do not take full advantage of the capabilities of all of the version control systems.  For example, it may be possible to do a "shallow" clone that does not pull in unnecessary history; some version control systems support a cache of some sort that can store history that might be common across multiple builds on the same slave.
    38 Currently all !WebStatus stuff is read-only (except triggering/canceling builds). It would be cool to be able to do some non-readonly things like:
     67Buildbot needs a consistent, easy, and configurable way for users to take advantage of these behaviors.
    40 - Add some description/notes to build at any time (for example notes to QA)
    41 - Making builds permanent (so this build will be never removed)
    42 - Some tags?
     69See also
     70 *
    44 == Slave-side repository caching ==
    46 Slaves often checkout/clone the entire repo (when doing full builds). This can take a lot of time and network traffic, both with CVS:es and DVCS:es.
    48 Having the option for a local clone on the slave side, for each repository used, would reduce network traffic to a minimum and increase checkout/update speed.
    50  * modern DVCS:es use hardlinks, so local clones are speedy and space-efficient.
    51  * repeated full cloning of DVCS repos can take a lot of time if the history is big.
    52  * CVCS:es might want to ease the load of the central server similarly. (f.ex. svnsync can be used to update a local "master" repo and the checkouts can then be done locally using file:// protocol)
    53  * buildbot try combined with DVCS:es might require the transfer of new commits (not just patches). These "temporary commits" are probably unwanted after the try build, and thus a clean clone is often wanted.
    54  * also, it might be easier to automate merge/rebase tests, just to see if integration is ok (while throwing away the resulting merge commit in the local repo)
     72=== Scope ===
     73This project begins with a significant design problem: figure out the common behaviors across a number of version control systems, and how to represent those behaviors to Buildbot's users.  Once that's done, you'll need to implement those new behaviors.  To allow for different amounts of available time over the summer, it will be smart to start with one version-control system, then tackle more if time allows.