Changes between Version 13 and Version 14 of DevelopmentPlan


Ignore:
Timestamp:
03/25/12 14:03:44 (14 months ago)
Author:
dustin
Comment:

new, clearer plan

Legend:

Unmodified
Added
Removed
Modified

  • DevelopmentPlan

    v13 v14  
    99 * Pluggable backends to suit different use-cases 
    1010 * Support distributed masters with no loss in functionality 
     11 
     12We will make a significant compatibility break at 0.9.0: 
     13 * Process objects (e.g., !BuildRequest) and status objects (e.g., !BuildStepStatus) are no longer part of the configuration interface 
     14 * Most custom subclasses (status listeners, schedulers, change sources, but notably not steps) will need to be rewritten to use the data api 
     15 * !WebStatus will be replaced with 'www', with a different configuration and customization interface. 
     16 * Various plugins like the status client and debug client will be replaced with implementations using different interfaces 
    1117 
    1218= Target Architecture = 
    … …  
    4854See the [http://trac.buildbot.net/roadmap Roadmap] for a mapping of the following to particular versions. 
    4955 
    50 = Status API = 
     56== Implement and use mq api == 
    5157 
    52 Before we can move status to a database, the interface to the status needs to be adjusted so that it's no longer synchronous.  The plan, at this point, is to design a well-defined API that recognizes both historical status (static) and builds in progress (dynamic and event-based).  This will be based significantly on the existing JSONRPC interface.  The API will be defined without regard to language, so that the same functions are defined for JSONRPC, Perspective Broker, asynchronous Deferreds, and any other interface. 
     58This entails implementing the mq api and converting what are now implemented as subcriptions to use the api -- specifically, changes, buildsets, and build requests.  This leaves the task of coordinating messages and db changes to the master, which operates directly against the db api and the mq api.  The !StatusReceiver interface is adapted so that the new messages are properly translated into method calls. 
    5359 
    54 Historical status will be represented with a get-heavy RPC API (e.g., get the results of this step, get the latest build for this builder), while activity will be represented with a message-queueing interface, where components listen for the events they want to hear about. 
     60== Include status in the db, mq, and data apis == 
    5561 
    56 Then, it remains to modify all of the status listeners to use this interface: 
     62The db api currently stops somwhere around source stamps and builds (and leaves a particularly ragged edge, sadly).  The API should be extended to cover everything currently described as "status" - builds, steps, logs, and so on - in the same API.  This does *not* entail moving that data into a database, but does entail a sufficiently flexible design to make that move later.  The process layer gets better-defined here, as it begins updating state via the data api, with corresponding messages sent via the mq api and state changes made via the db api. 
    5763 
    58 == Web Status to new API == 
     64Because the db api is asynchronous, this step will break all of the existing users of the (mostly synchronous, unspecified, and horribly convoluted) status interface.  So this is a moment when a release is not possible.  However, the rest is downhill. 
    5965 
    60 Rather than redevelop the existing, templated web status using a new Python API (which would entail a complete rewrite), the new web status will be written in !JavaScript, which can be served statically -- either by Buildbot or any other web service.  It will use the JSONRPC interface to get the data it needs and render it in the browser.  This is a critical step before beginning work on the statusdb, as the existing template system makes heavy use of synchronous status-fetching methods.  Open questions: 
    61  * what !JavaScript library can we use?  How does licensing limit our choices? 
     66== Implement the www layer == 
    6267 
    63 == IRC == 
     68This means implementing the REST API and methods for consuming messages, along with some client-side !JavaScript to make it easy to use there. 
    6469 
    65 The IRC plugin will need to be rewritten, as well.  It's undecided as to whether this rewrite will be strictly in-process, using the Twisted version of the status API, or potentially run in another process, using one of the remote versions of the API. 
     70== Implement the web UI == 
    6671 
    67 == Notifiers == 
     72Related, but in a separate project, we'll need to build a good user experience on the web. 
    6873 
    69 The !MailNotifier and !TinderboxMailNotifier will both need to be rewritten.  There's no good reason to move a notifier out of process, so these will likely remain within the buildmaster. 
     74== Rewite status plugins == 
    7075 
    71 = statusdb = 
    72  * #24 
     76The remaining status plugins will need to be rewritten to use the data api. 
    7377 
    74 We are all waiting with bated breath for the status pickles to die a firey death.  Once the status API is complete, we are free to change out the data-storage backend through the usual database migration system, reading each and every build pickle into memory and committing it to the database. 
     78= Not Included = 
     79 
     80There are a number of other improvements that can be made in parallel with the above, but are not related: 
     81 * message-based master/slave communication 
     82 * declarative configuration