wiki:DevelopmentPlan

This page sketches the future of Buildbot. This is an open-source project with no full-time developers, so no timeline is in place. Instead, this serves as a roadmap for important projects so that we can focus our efforts on the next step or two. Of course, along the way we will fix lots of defects and make several releases in the process.

Goals

  • Efficient, scalable operation
  • Provide well-defined APIs for users to build on
  • Store all state in a database
  • Real-time updates as builds proceed
  • Modern, fast web interface
  • Pluggable backends to suit different use-cases
  • Support distributed masters with no loss in functionality

We will make a significant compatibility break at 0.9.0:

  • Process objects (e.g., BuildRequest) and status objects (e.g., BuildStepStatus) are no longer part of the configuration interface
  • Most custom subclasses (status listeners, schedulers, change sources, but notably not steps) will need to be rewritten to use the data api
  • WebStatus will be replaced with 'www', with a different configuration and customization interface.
  • Various plugins like the status client and debug client will be replaced with implementations using different interfaces

Target Architecture

buildbot-architecture.png Buildbot's per-master architecture is illustrated at the right. From bottom to top, it operates as follows. (note that the Inkscape source for the image above is in https://github.com/buildbot/buildbot/tree/master/media)

db

The db layer implements a pluggable, abstracted, persistent storage layer. All masters in a cluster have access to the same persistent storage, although Buildbot does not make any strong assumptions about the timing of updates to that storage. For example, a replicated MySQL database may have up to a few seconds' lag between a write from one master and a read of that data on another master.

The db api presented by this layer is a Python-only, asynchronous API, and is very closely tied to the data being stored.

mq

The mq layer implements a pluggable message queueing system. Depending on your background, you can think of this as a single message bus supporting pattern-matching against structured message topics, or in AMQP terms a single topic exchange. The mq layer itself is content-agnostic. The mq layer interacts with the db layer to update per-master clocks, which are used by the data layer to provide consistency guarantees.

The mq api simply allows callers to publish and consume messages.

data

The data layer combines access to stored state and messages, ensuring consistency between them, and exposing a well-defined API that can be used both internally and externally. Using caching and the clock information provided by the db and mq layers, this layer ensures that its callers can easily receive a dump of current state plus updates to that state, without missing or duplicating updates.

The data api accomplishes a few things:

  1. read-only access:
    • getters - fetching data from the db API, and
    • subscriptions - subscribing to messages from the mq layer;
    both are available via a Python interface and wrapped in an HTTP interface.
  2. control: authenticated requests for actions in the buildmaster: forcing a build, cancelling a build request, shutting down cleanly, etc.
  3. updates: a Python API internal to the buildmaster which allow direct updates to state while sending appropriate messages. This is how process code records new changes, builds, steps, and so on.

See DevelopmentPlan-DataApi for more detail.

status listeners

This layer contains components that consume state changes and present that information as desired. The prototypical example is the IRC client. If implemented in the Buildmaster process, these components can use the Python data API. Otherwise, they can be built atop the language-agnostic APIs exposed by the www layer.

www

This layer exposes the data API externally, via typical web technologies like a REST API, WebSockets?, comet, and so on. It also contains and serves the static content that comprises the Buildbot web UI, although that content can just as well be served by a dedicated application like Apache httpd or nginx. This layer is responsible for web-based authentication of access to the control section of the data api.

See DevelopmentPlan-Www for more detail.

process

This little box holds all of the interesting parts of Buildbot. Change sources, Schedulers, Builders, Builds, Steps, and so on are all part of the process layer. In fact, these are all distinct components within this layer: change sources use the rw section of the data api to add changes, and do not interact directly with schedulers. Where necessary, schedulers use the ro section to learn about new changes, and then use the rw section to schedule new builds. Likewise, builders are alerted to new build requests by the data layer, and feed the results of the subsequent builds back into the data layer.

Implementation

Most of the above will need to be implemented before a release can be made - which may take a while! The plan is to work on this project while continuing the 0.8.x series, and release 0.9.0 when the new design is implemented and stable, with performance improvements and new features coming in subsequent 0.9.x releases.

Progress on the implementation is tracked in-repo at https://github.com/buildbot/buildbot/blob/nine/README.md.

Last modified 4 years ago Last modified on Mar 20, 2013, 7:47:27 PM

Attachments (1)

Download all attachments as: .zip