wiki:ProjectIdeas

Version 40 (modified by tom.prince, 2 years ago) (diff)

Update master-slave project to mention amp+ssh.

If you're interested in working on any of these ideas, or have ideas of your own that you'd like to work on, get in touch! See the links on Development, or email the project maintainer directly - dustin@….

GSoC 2012

The projects on this page are, of course, open to anyone who wants to contribute, but the page is designed specifically for potential Google Summer of Code students. Keep in mind that we have three goals for GSoC:

  • We gain new contributors to Buildbot
  • You learn about open-source development by doing it
  • Users get amazing new Buildbot features

We'd like to achieve all three goals, but the world is messy, and predictions are hard. If you finish the summer having learned a lot and become a lifelong contributor to Buildbot, but without wrapping up a user-visible feature, that's still OK.

Applications should follow the form given in GsocApplicationTemplate.


Your Idea Here!

We are open to any ideas that will make Buildbot better - think of this page as a set of ideas to get you started.


JavaScript Frontend

Owner: Pierre Tardy

Currently, the web frontend is rendered synchronously by the Buildmaster with jinja2 templates, based on data from the database and from build pickles. This has two disadvantages:

  • the entire Buildmaster is blocked while the page is being rendered, which can be a significant time as build pickles are slow to laod
  • once SQLAlchemy is fully implemented (which will happen before GSoC 2011 begins), database queries cannot be made synchronously

The preferred solution at this point is to write the web frontend in browser-side JavaScript, using the existing JSON interface to gather the necessary data.

Scope

Work on this project would begin with a simple proof-of-concept page, selection of a JavaScript framework, and the necessary Python code to support all of it. This will involve a lot of discussion about problems that may come up later in the project, and planning ahead for them. Once this has been reviewed and accepted, you can go page-by-page through the Buildbot interface, rewriting each (or even parts of each) in JavaScript.

The JS frontend will talk to the JSON REST service provided by the Buildmaster. Much of this service is already implemented, but it will need to be expanded and improved, which will require some Python programming and learning a bit about Twisted Python.

Issues to consider:

  • testing - how will we test the JS?
  • compatibility - what browsers will we support? May probably want to focus on latest version of browsers but care about mobile browsers too.
  • configuration - how can we configure the frontend using parameters in the Buildmaster's configuration file?
  • security - how can we protect information from being seen by those who should not see it? How can we limit who can start or stop builds?
  • licensing - if we redistribute a JS framework, its license must be compatible with Buildbot's
  • documentation - how can we do a better job of documenting the web frontend for others to use?

Windows Process Management

Owner:

The windows tag documents a number of bugs about starting and killing processes on Windows. Buildbot uses Twisted's process-handling code, so these bugs may be fixed by either re-implementing better support directly in Buildbot, or by patching Twisted's process-handling code.

See also:

Scope

To do well with this project, you would need to bring a lot of the Windows experience that is lacking in the Buildbot development community. Assuming you're familiar with Windows APIs and accessing them from Python, this bug would entail

  • writing test cases to reproduce bugs users have seen
  • interacting with the Twisted community to design solutions that can be merged upstream
  • implementing portable fixes to those bugs
  • documenting them

New Master-Slave Protocol

Owner: tomprince

Buildbot currently uses Twisted Python's perspective broker to communicate with slaves. This is a remote procedure call library that operates on TCP connections, and has a few disadvantages:

  • It is Python-only, so slaves must be implemented in Python
  • It keeps the TCP connection open for many hours, and does not handle connection failure well
  • The RPC model is complex and does not map well to the operations Buildbot performs
  • The RPC implementation is inefficient and imposes some arbitrary limits on Buildbot's flexibility.

Scope

Here be dragons! Several people have attempted this before:

either of these may be a great starting point for this project, but at any rate this is a challenging project that will require a lot of thoughtful design work. The current plam is to use amp over ssh as the default slave protocol.

It's probably worth looking at off-the-shelf Message Queuing projects like zeromq, RabbitMQ or Celery.

The best approach is to find a way to get some working code put together quickly, while still allowing Buildbot to do everything its users expect. For example, if you can write your new protocol such that Buildbot can use perspective broker or your protocol, but you only get as far as a simple slave that can run echo hello world, that's great! We can ship that code, and someone else can pick up where you've left off to add more capabilities to your protocol -- assuming your design does not make that impossible.

It would be great to select a communication protocol that is not Python-specific, so that a non-Python slave could be used to run Buildbot on more limited hardware (e.g., mobile devices).


Full integration for users

Owner:

Buildbot deals with users in many ways - through version control, through the web, from IRC, from the command line for buildbot try and buildbot sendchange, and so on. In GSoC 2011, Derek Hurley added general support for correlating users across these interfaces, but the work remains unfinished. This project would involve adding the additional configuration and support for interacting with users:

  • proper calculation and display of "blamelists" for changes
  • authentication for submitting jobs and other commands (IRC, web, command-line)
  • automatic correlation of user attributes based on rules specific to the organization (e.g,. "All email addresses are the subversion username @mycompany.com" or "Use email addresses from git commits in the absence of an existing email address" or "IRC nicknames match bzr usernames")
  • stored user preferences for notifications

Relevant bugs are tagged with users.

scope

This project has the advantage of starting from a solid base - there will be more coding than design here. It also breaks down nicely into individual tasks that can be completed one after the other, allowing you to scale the scope of the project up or down as the summer proceeds.


Complete Master-Side Source Steps

Owner:

GSoC 2011 included a project to add master-side source steps. These are a new implementation of steps that have historically housed their logic in the slave's code, putting that logic in the master instead. There were a few key pieces of functionality omitted from the new master-side source steps, and until that functionality is complete Buildbot is stuck maintaining *both* - an awkward position.

The relevant bugs are tagged with master-side-source-steps.

scope

This project would involve surveying the missing functionality, implementing it, and adding tests and documentation. This involves relatively little design work, but a good bit of hands-on debugging and coding.


Declarative Configuration (#2218)

Owner:

Buildbot's configuration has historically been a Python file that Buildbot executes. The result of that execution is then used to set up the master. This assumes users can write valid Python code, and encourages people to do all manner of fragile or non-maintainable things. See #2218 for more.

scope

This project will involve some careful thought about how users interact with Buildbot, and how we can maintain backward compatibility while providing users with a forward migration path. Once that's complete, the modifications to the code will be fairly minimal, and mostly limited to master/buildbot/config.py (with associated tests). However, the change will require alteration -- a rewrite, really -- of the entire "Configuration" section of the manual; this will be the most time-consuming (but critical) part of this project.

examples