wiki:ProjectIdeas

Version 28 (modified by dustin, 8 years ago) (diff)

--

GSoC 2011

Table of Contents

    If you're interested in working on any of these ideas, or have ideas of your own that you'd like to work on, get in touch! See the links on Development, or email the project maintainer directly - dustin at v.igoro.us or maruel at chromium.org.

    Keep in mind that we have three goals for GSoC:

    • We gain new contributors to Buildbot
    • You learn about open-source development by doing it
    • Users get amazing new Buildbot features

    We'd like to achieve all three goals, but the world is messy, and predictions are hard. If you finish the summer having learned a lot and become a lifelong contributor to Buildbot, but without wrapping up a user-visible feature, that's still OK.


    Your Idea Here!

    We are open to any ideas that will make Buildbot better - think of this page as a set of ideas to get you started.


    JavaScript Frontend

    Owner: maruel

    Currently, the web frontend is rendered synchronously by the Buildmaster with jinja2 templates, based on data from the database and from build pickles. This has two disadvantages:

    • the entire Buildmaster is blocked while the page is being rendered, which can be a significant time as build pickles are slow to laod
    • once SQLAlchemy is fully implemented (which will happen before GSoC 2011 begins), database queries cannot be made synchronously

    The preferred solution at this point is to write the web frontend in browser-side JavaScript, using the existing JSON interface to gather the necessary data.

    Scope

    Work on this project would begin with a simple proof-of-concept page, selection of a JavaScript framework, and the necessary Python code to support all of it. This will involve a lot of discussion about problems that may come up later in the project, and planning ahead for them. Once this has been reviewed and accepted, you can go page-by-page through the Buildbot interface, rewriting each (or even parts of each) in JavaScript.

    The JS frontend will talk to the JSON REST service provided by the Buildmaster. Much of this service is already implemented, but it will need to be expanded and improved, which will require some Python programming and learning a bit about Twisted Python.

    Issues to consider:

    • testing - how will we test the JS?
    • compatibility - what browsers will we support? May probably want to focus on latest version of browsers but care about mobile browsers too.
    • configuration - how can we configure the frontend using parameters in the Buildmaster's configuration file?
    • security - how can we protect information from being seen by those who should not see it? How can we limit who can start or stop builds?
    • licensing - if we redistribute a JS framework, its license must be compatible with Buildbot's
    • documentation - how can we do a better job of documenting the web frontend for others to use?

    Windows Process Management

    Owner: djmitche

    The windows tag documents a number of bugs about starting and killing processes on Windows. Buildbot uses Twisted's process-handling code, so these bugs may be fixed by either re-implementing better support directly in Buildbot, or by patching Twisted's process-handling code.

    See also:

    Scope

    To do well with this project, you would need to bring a lot of the Windows experience that is lacking in the Buildbot development community. Assuming you're familiar with Windows APIs and accessing them from Python, this bug would entail

    • writing test cases to reproduce bugs users have seen
    • interacting with the Twisted community to design solutions that can be merged upstream
    • implementing portable fixes to those bugs
    • documenting them

    Multi-Repository Builds and Build Coordination

    Owner: djmitche

    NOTE: this project is actively being worked on by Harry Borkhuis

    As implemented right now, a single Buildmaster can build multiple projects, but each project must be built from a single source code repository, and builds of the projects must not interact. Increasingly, though, users want to build projects that require source from multiple repositories, and where a checkin to any of those repositories should trigger a build. Similarly, users often want to coordinate multiple projects, for example a web frontend and backend server that are built separately but must be tested together or packaged into a single installer.

    See also:

    Projects and Scope

    Fixing this is a big, ongoing task that cannot be completed in a summer, so this project would involve *improving* support for these sorts of configurations. Part of the project (and proposal), then, is defining what problem you'd like to solve. We have discussed a few ideas that might get you started:

    • Sourcestamps that point to multiple repositories and revisions in those repositories. Builds would then specify a way to check out the given revision of each repository before beginning the compile and test steps. Designing these would involve some significant changes to the database schema, as well as a solid plan for backward compatibility. Because changes would be required on the master and slave, you would need to think about compatibility between different versions of master and slaves, as well. If possible, it would be smart to concentrate on only one version-control system for a summer project, so that others could fill in the blanks later.
    • Source managers that know everything about a particular version control repository. A source manager, for example, could determine the changes that were made between two source stamps (revisions), or provide the information required to check out a particular revision from a repository. Source managers, too, could be implemented for a single version control system, so that others can finish the work for the remaining systems.
    • "Gaggles" of builds. Right now, it is very difficult to wait for a number of different builds to finish - for example, if you need the Windows and Linux builds to finish so that you can compress them into a CD image. The most difficult part is to determine when two builds "go together", even if they occur at different times. If we can group builds into gaggles, then it's easy to wait for, for example, all of the Linux and Windows builds in a particular gaggle to finish before starting the CD-burning build. This project would involve a lot of design discussions with Buildbot users, then some database and code changes to manage gaggle identifiers. Finally, some easy-to-use synchronization steps and documentation of how to use them would make this available to Buildbot's users.

    New Master-Slave Protocol

    Owner: djmitche or maruel

    Buildbot currently uses Twisted Python's perspective broker to communicate with slaves. This is a remote procedure call library that operates on TCP connections, and has a few disadvantages:

    • It is Python-only, so slaves must be implemented in Python
    • It keeps the TCP connection open for many hours, and does not handle connection failure well
    • The RPC model is complex and does not map well to the operations Buildbot performs

    Scope

    Here be dragons! Several people have attempted this before:

    either of these may be a great starting point for this project, but at any rate this is a challenging project that will require a lot of thoughtful design work. It's probably worth looking at off-the-shelf Message Queuing projects like zeromq, RabbitMQ or Celery.

    The best approach is to find a way to get some working code put together quickly, while still allowing Buildbot to do everything its users expect. For example, if you can write your new protocol such that Buildbot can use perspective broker or your protocol, but you only get as far as a simple slave that can run echo hello world, that's great! We can ship that code, and someone else can pick up where you've left off to add more capabilities to your protocol -- assuming your design does not make that impossible.