Opened 23 months ago

Last modified 12 months ago

#3518 new enhancement

Add support for worker pools

Reported by: thalin Owned by:
Priority: minor Milestone: 0.9.+
Version: master Keywords: master-slave slave


Configuring each worker in a config file can be painful when adding many identical workers, such as in an Amazon or Docker environment. As an administrator of a private Buildbot instance, I would like to be able to configure a scalable group of workers with one configuration directive, such as:

c['workers'] = [worker.workerPool("poolname", "password", ...), ...]

and have N workers able to register using the pool name and password. Then in builds, I should be able to specify just the pool name and have any worker from that pool be able to execute those builds.

This would allow blind scaling of workers, which is useful in environments such as Docker Compose, where adding new workers would become as simple as:

docker-compose scale worker=5

Change History (3)

comment:1 Changed 23 months ago by tardyp

I like the feature.

It has interesting consequences in the core though. Needs to make the list of workers dynamic, or probably simpler, reconfigure automatically the worker-manager with an autoscaling algorithm.

comment:2 Changed 23 months ago by dustin

  • Milestone changed from undecided to 0.9.+
  • Type changed from undecided to enhancement

This would be a great feature, especially for those using the latent worker support.

It's not immediately clear how best to implement this, particularly in terms of efficiency (we have things that are O(n2) in the number of workers). So, some prototyping is in order. It would be great to land this after the dust from 0.9.0 has settled.

Last edited 23 months ago by sa2ajj (previous) (diff)
Note: See TracTickets for help on using tickets.