Opened 7 years ago

Last modified 5 years ago

#2644 new enhancement

Add more MQ backends

Reported by: dustin Owned by:
Priority: critical Milestone: 0.9.+
Version: Keywords: mq


The MQ layer currently only has a simple (single-master) implementation. We should have some or all of

  • AMQP
  • AMP-based master-to-master communication (full mesh, with every master talking to every other master, or something smarter)
  • ZeroMQ

Change History (7)

comment:1 Changed 7 years ago by dustin

Actually, Kombu supports all of these things including an in-memory version and a DB-based version, and is a simple Python dependency. We should probably just require Kombu and use its functionality directly, replacing the entire MQ layer.

comment:2 Changed 7 years ago by dustin

  • Keywords mq added

comment:3 Changed 6 years ago by tardyp

This project idea has been worked on during gsoc 2014. Even if this was a failed project, there has been some good discussion and design hints by djmitche on the mailing list:

comment:4 Changed 6 years ago by tardyp

Good MQ usecase introduction is described here:

Technics from this document are relevent to buildbot:

3) pub/sub: most of the bb usecases are using this (

  • Inform other masters of new buildrequests
  • Inform status plugins of updates in the builds
  • Inform Web UI of changes in any of the data

6) RPC: ( In order to avoid web masters to have db creds in write mode, long term plan is to use mq RPC technic for control APIS

  • force scheduler
  • build rebuild
  • build stop
  • buildrequest cancelation

comment:5 Changed 6 years ago by dustin

  • Priority changed from major to critical
  • Summary changed from [nine] Add more MQ backends to Add more MQ backends

We need this for the 0.9.0 release -- the simple implementation in place now only works for a single-master configuration.

comment:6 Changed 5 years ago by sa2ajj

How do we need this for 0.9.0 release? 0.9.0 beta can be deployed without additional MQ backends, do we really need more backends?

comment:7 Changed 5 years ago by dustin

  • Milestone changed from 0.9.0 to 0.9.+

Now that we have support, no, this is not a 0.9.0 blocker.

Note: See TracTickets for help on using tickets.