wiki:ProjectIdeas

If you're interested in working on any of these ideas, or have ideas of your own that you'd like to work on, get in touch! See the links on Development, or email the project development mailing list.

GSoC 2016

The projects on this page are, of course, open to anyone who wants to contribute, but the page is designed specifically for potential Google Summer of Code students. Keep in mind that we have three goals for GSoC:

  • We gain new contributors to Buildbot
  • You learn about open-source development by doing it
  • Users get amazing new Buildbot features

We'd like to achieve all three goals, but the world is messy, and predictions are hard. If you finish the summer having learned a lot and become a lifelong contributor to Buildbot, but without wrapping up a user-visible feature, that's still OK.

Applications should follow the form given in GsocApplicationTemplate.

Open Ideas

Ticket Summary Owner
#2966 New web plugin idea: a health indicator
Description

I like to extract as much useful indicator from my builds as possible (time, but also amount of warnings, and such ...)

It would be cool to have a web plugin that could print the evolution of my indicators over time ! (Of course, I would have to configure which indicator I want to see plotted, maybe the kind of plot, and so on ...)

#3452 [AngularJS Project] Add E2E tests
Description

We need some capability to run end-to-end tests of the AngularJS frontend to avoid UI regressions and other issues that are not obvious in unit tests.

This should be done with the normal angularJS methodology, with protractor: https://docs.angularjs.org/guide/e2e-testing

What is necessary for e2e tests is:

  • create a real buildbot master with a real configuration, which contains configuration for major use cases involving the UI. e.g force schedulers, authentication, etc
  • Have protractor scenarios which acts like a user would, and verify that the UI behaves as expected.
  • User authenticates in the UI
  • User go to the list of builds for a builder
  • User click on the forcescheduler button
  • User adds some input in the modal dialog
  • build is started, and there are updates in the UI

The idea should be to hook up the tests in the metabuildbot, and allow motivated developers to run them manually, but they need not be as easy to run as 'trial buildbot.test'.

Documentation on what are the steps to run that (probably easiest is to setup the deps in a dockerfile)

#3148 [AngularJS project] Optimised communication between backend and frontend tothandras
Description

Communication between backend and frontend is a tricky problem.

Web browser must get the current state of an object, and then register to a message queue to automatically update its internal model.

A good prior art for such system is firebase: https://www.firebase.com/docs/

Buildbot nine implement similar ideas in buildbotServices, but has a number of bottlenecks.

  • Based on restangular, the data is mixed-in with restangular methods, which prevents to walk over all attributes of a dictionary
  • There are race conditions between initial http get, and message queue updates, resulting in out of sync model
  • The data model being fully generic prevents adding some decoration to the model depending on the type. Example is email parsing of change_owner is made in several place (search emailRegex in the code)
  • Too much http requests are needed to make the system work

Student will analyse current implement implementation, and propose changes to address those issues. Probably a full rewrite will be necessary.

#2044 Make buildbot high-scores board
Description

The idea is to have a web application which aggregates various buildbot activity into one place and assigns a total score to each contributor. It should be a fun way to motivate contributors and help them to compare their level of effort with others.

This would not be a part of the Buildbot application, but a separate service written and maintained by the Buildbot developers.

refs:

Related:

#896 Replace try with a client for the force scheduler verm
Description

We have a "try" client defined which looks for a patch in a local repository on a developer's system, patches it up, and submits it to Buildbot. It's very ad-hoc, isn't very configurable, and hasn't gotten a lot of love.

Now that we have a force scheduler, the better solution is for the try client to talk to a force scheduler, preferably using HTTP. The client should also capture and send e-mail address of the submitter so that notifications can be sent.

This allows lots of flexibility on the master side: users can set up as many force schedulers as they like, with different builders, properties, permissions, and so on.

scope

To make this project a summer's worth, it should aim to implement:

  • a new try client
  • improved support for DVCS's, where patches aren't required, but where the client must send a repository
  • support in the master for sending force-build parameters to a force scheduler via an HTTP rest call
  • support for patches in the force scheduler
  • per-scheduler authentication/authorization
  • migration information for users of the existing try functionality
#3453 [AngularJS Project] Improve AngularJS unit testing framework
Description

The unit tests for Buildbot's Angular components need to be expanded. That means more tests, but it also means more support code for those tests. In particular, a realistic means of simulating the Buildbot server backend. Ideally that would be a fake backend, and one that is guaranteed to match the real thing by use of RAML or JSON-Schema or similar API-description techniques.

This project would involve picking the API-description technique, applying it in the Python code, and then writing the fake backend in CoffeeScript. And writing tests that use the backend, to show that it works.

There are other options here -- perhaps running the Buildbot master in a "test backend" mode that allows extra API calls. The end result should be a way to realistically test Angular components against the Buildbot Data API.

#2590 Use Alembic instead of SQLAlchemy-Migrate
Description

Migrate has some compatibility problems - in particular, it doesn't work with SQLAlchemy >= 8.0. Its API is a little unpredictable, too, and it doesn't do a very good job of masking differences between dialects. Its linear numbering scheme is also problematic when developing on topic branches, as a merge of branches with overlapping DB version numbers will need some substantial changes to put them in linear order.

Alembic seems to be the fix to all that, although that's as far as we've gotten.

This would make a decent, if small, Google Summer of Code project.

Scope

The goal of this project is clear: replace the functionality of SQLAlchemy-Migrate with equivalent functionality implemented with Alembic. A GSoC application should also address:

  • Compatibility - users with current versions of Buildbot must be able to upgrade to Alembic
  • Testing - when database migrations fail, users lose data. The current migrations are thoroughly tested to ensure they do the right thing on all supported databases, and the new implementation must do the same.
#2649 i18n -- Internationalization
Description

The Web UI should be localized. AngularJS ecosystem has ngTranslate project, which makes i18n relatively easy.

scope

This project would require

  • Learning about AngularJS and ngTranslate and how they are applied to an application
  • Applying ngTranslate to the Buildbot JS source code
  • Documenting how to correctly use ngTranslate in new code
  • Documenting the process for adding or updating a localization
  • Testing translation
#3147 [AngularJS project] Use angular dashboard framework for nine homepage
Description

Angular dashboard framework provides nice user interface for letting the user customize their homepage.

show page of this project: http://sdorra.github.io/angular-dashboard-framework

It is compatible with the boostrap CSS framework that we are based on, and has a nice api for defining new widgets.

This project is about integrating this framework in buildbot, and creating a few widgets.

Example widgets could be:

  • Status for a builder (last build)
  • List of building builds
  • List of disconnected slaves
  • Stats about people responsible of failed builds
  • Stats about people responsible of fixing failed builds
#2890 Improve support for Gerrit
Description

There's a number of things that could be improved for Gerrit support.

(And I'd need to create a ticket for this particular problem: force the version (if an installation is prepared incorrectly, Gerrit fails to report its version: might fail certain actions).)

List of relevant tickets

defect type tickets:

#1733
GerritStatusPush: missing retrying mechanism for status updates
#2543
GerritChangeSource should ignore incoming ref-updated events to refs/meta/config branch
#2755
gerritStartCB don't work
#3273
GerritChangeFilter support for branch filtering is broken
#3649
GerritChangeFilter doesn't trigger a builder if a push is done without a gerrit review

enhancement type tickets:

#1725
GerritStatusPush: add filters on builderName, project, branch

project-idea type tickets:

#2890
Improve support for Gerrit

support-request type tickets:

#3428
Multiple GerritStatusPush after reconfiguring buildbot

#2438 Windows Process Management
Description

The windows tag documents a number of bugs about starting and killing processes on Windows. Buildbot uses Twisted's process-handling code, so these bugs may be fixed by either re-implementing better support directly in Buildbot, or by patching Twisted's process-handling code.

See also:

Scope

To do well with this project, you would need to bring a lot of the Windows experience that is lacking in the Buildbot development community. Assuming you're familiar with Windows APIs and accessing them from Python, this bug would entail

  • writing test cases to reproduce bugs users have seen
  • interacting with the Twisted community to design solutions that can be merged upstream
  • implementing portable fixes to those bugs
  • documenting them

Related Bugs

Ticket Summary Owner
#387 TerminateProcess fails in buildbot_service.py
#456 Slave's ShellCommand and msys are broken even
#1061 Buildbot locks downloaded files
#1871 Goofy quoting/space handling in buildbot_service.py
#2073 buildbot is unable to delete subdirectories of the build dir that have non-ASCII names under Windows davidsarah
#2176 buildslave hangs trying to kill process after "1200 seconds without output" Callek
#2449 Buildbot fails to capture output
#2878 Windows escaping seem to produce problems
#2885 Windows: rmdirRecursive() fails if a contained file is opened
#2923 "simple" builds fail because Buidlbot is not installed
#2936 Buildbot - Traceback while polling for changes issue
#3052 "File is being used by another process"
#3251 'Git' command fails to clobber local repository when repository has a corrupt index file
#2108 BuildBot uses incorrect method for default location of Visual Studios
#1754 VC9 compile step assumes PlatformSDK is installed
#2293 Simplify MSYS+buildslave integration
#2372 contrib/svn_buildbot.py doesn't work on Windows
#3512 [windows] pid file missing

#2692 Auto-generate some documentation
Description

Lots of the Buildbot documentation is manually maintained to correspond to the code: buildstep arguments, data API descriptions, DB APIs, and so on.

It'd be great to have some means of automatically verifying or even generating that documentation. http://dexy.it/ looks like a useful general tool for building this kind of thing.

Scope

This project would involve writing code and configuration to automate the documentation-generation process. This should take Buildbot's source code as input (preferably parsed by Python, rather than with regular expressions or some other text-mode process) and generate rst output that can then be processed using Sphinx.

A good GSoC application will identify several areas of the documentation that can be converted to automatic generation. These should become milestones in the project schedule.

#1788 MQ options
Description

When Buildbot adopts an MQ framework, there are a number of choices. Among them:

The first is already implemented in 'nine', but we need more.

The more difficult part of this project is to ensure that messages are only delivered after the corresponding database changes are visible.

The issue is a race condition between message passing and database replication. Imagine you have a large Buildbot installation with several replicated MySQL servers and a redundant RabbitMQ cluster. One buildbot master writes changes to a build to the database (an UPDATE operation), then sends a message describing the change. On another master, some service gets the message and queries a different MySQL server to see the build's new status. If the message arrives before the database replication occurs, then this master will see stale data. That will lead to a lot of subtle, rare bugs.

scope

This project would involve separate tasks:

  • implement one or more of the above MQ plugins (mostly just coding)
  • solve the data-ordering problem (requiring some CS theory)

relevant bugs

Your Idea Here!

We are open to any ideas that will make Buildbot better - think of this page as a set of ideas to get you started.

Last modified 13 months ago Last modified on Feb 14, 2016, 1:53:29 PM