Changes between Version 19 and Version 20 of DevelopmentPlan


Ignore:
Timestamp:
Mar 20, 2013, 7:47:27 PM (5 years ago)
Author:
dustin
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • DevelopmentPlan

    v19 v20  
    3333The data layer combines access to stored state and messages, ensuring consistency between them, and exposing a well-defined API that can be used both internally and externally.  Using caching and the clock information provided by the db and mq layers, this layer ensures that its callers can easily receive a dump of current state plus updates to that state, without missing or duplicating updates.
    3434
    35 The data api is divided into four components:
     35The data api accomplishes a few things:
    3636
    37  1. ro ("read only"), which allows reading any state and subscribing to any messages, made of two components --
     37 1. read-only access:
    3838   * getters - fetching data from the db API, and
    3939   * subscriptions - subscribing to messages from the mq layer;
    40  1. control, which allows state to be changed in specific ways by sending appropriate messages (e.g., stopping a build); and
    41  1. rw ("read/write"), which allows direct updates to state while sending appropriate messages
    42 
    43 The ro section is exposed everywhere.  Access to the control section should be authenticated at higher levels, as the data layer does no authentication.  The rw section is for use only by the process layer -- all external interaction with that layer should be done via the rw section.
     40  both are available via a Python interface and wrapped in an HTTP interface.
     41 1. control: authenticated requests for actions in the buildmaster: forcing a build, cancelling a build request, shutting down cleanly, etc.
     42 1. updates: a Python API internal to the buildmaster which allow direct updates to state while sending appropriate messages.  This is how process code records new changes, builds, steps, and so on.
    4443
    4544See [wiki:DevelopmentPlan-DataApi] for more detail.
     
    4948
    5049== www ==
    51 Technically, the www layer is a status listener, but it is complex enough to deserve its own box in the diagram.  This layer exposes the data API externally, via typical web technologies like a REST API, WebSockets, comet, and so on.  It also contains and serves the static content that comprises the Buildbot web UI, although that content can just as well be served by a dedicated application like Apache httpd or nginx.  This layer is responsible for web-based authentication of access to the control section of the data api.
     50This layer exposes the data API externally, via typical web technologies like a REST API, WebSockets, comet, and so on.  It also contains and serves the static content that comprises the Buildbot web UI, although that content can just as well be served by a dedicated application like Apache httpd or nginx.  This layer is responsible for web-based authentication of access to the control section of the data api.
    5251
    5352See [wiki:DevelopmentPlan-Www] for more detail.
     
    6059Most of the above will need to be implemented before a release can be made - which may take a while!  The plan is to work on this project while continuing the 0.8.x series, and release 0.9.0 when the new design is implemented and stable, with performance improvements and new features coming in subsequent 0.9.x releases.
    6160
    62 The rough task breakdown is:
    63 
    64 == Implement and use mq api ==
    65 
    66 This entails implementing the mq api and converting what are now implemented as subcriptions to use the api -- specifically, changes, buildsets, and build requests.  This leaves the task of coordinating messages and db changes to the master, which operates directly against the db api and the mq api.  The !StatusReceiver interface is adapted so that the new messages are properly translated into method calls.
    67 
    68 == Implement the data api ==
    69 
    70 Most components of the data api are thin wrappers around other APIs (db and mq), although the data api implementation provides a good home for the complex functions to add and notify about buildsets, changes, and so on -- the update component.
    71 
    72 == Include status in the db, mq, and data apis ==
    73 
    74 The db api currently stops somewhere around source stamps and builds (and leaves a particularly ragged edge, sadly).  The API should be extended to cover everything currently described as "status" - builds, steps, logs, and so on - in the same API.  This does *not* entail moving that data into a database, but does entail a sufficiently flexible design to make that move later.  The process layer gets better-defined here, as it begins updating state via the data api, with corresponding messages sent via the mq api and state changes made via the db api.
    75 
    76 Because the db api is asynchronous, this step will break all of the existing users of the (mostly synchronous, unspecified, and horribly convoluted) status interface.  So this is a moment when a release is not possible.  However, the rest is downhill.
    77 
    78 == Implement the www layer ==
    79 
    80 This means implementing the REST API and methods for consuming messages, along with some client-side !JavaScript to make it easy to use there.
    81 
    82 == Implement the web UI ==
    83 
    84 Related, but in a separate project, we'll need to build a good user experience on the web.
    85 
    86 == Rewrite status plugins ==
    87 
    88 The remaining status plugins will need to be rewritten to use the data api.
    89 
    90 == Later ==
    91 
    92 This is where a release becomes possible.  Subsequent work:
    93  * caching in the data api
    94  * synchronization of state and events in the data api
    95  * move status to the db api
    96 
    97 = Not Included Here =
    98 
    99 There are a number of other improvements that can be made in parallel with the above, but are not related:
    100  * message-based master/slave communication. See [wiki:MasterSlaveCommunication].
    101  * declarative configuration
     61Progress on the implementation is tracked in-repo at https://github.com/buildbot/buildbot/blob/nine/README.md.