Opened 5 years ago

Last modified 3 years ago

#2959 new defect

Support a single config across the cluster, or something like that

Reported by: Ben Owned by:
Priority: major Milestone: 0.9.+
Version: master Keywords: multimaster


The www interface is showing all builders that ever where there. This is a difference with eight were only the 'active' builders were displayed on the web interface. I believe the latter is wished.

Change History (13)

comment:1 Changed 5 years ago by dustin

  • Keywords nine www removed
  • Milestone changed from 0.9.0 to 0.9.+

This is a little tricky -- we don't want builders to "disappear" if the master that happens to be hosting them goes down. Although we could do that now, with a rather complex set of queries.

I suspect that the better plan will be to have builders disappear using the same kind of "horizon" cleanup we do for logs and builds.

comment:2 Changed 5 years ago by Ben

Provided this 'horizon' cleanup would be implemented in 0.9.+, this would then, if I understand correctly, prevent me to look at builders where build didn't happen for some time, even if the builders are still 'active' (defined in my config).

Anyway, the trouble I'm having now, is that at each reconfig where I renamed a builder, the old one stay there. Those were previous tests from me, and have no value anymore once the renamed builder is there.

I need a way to cleanup my interface, and anyone who experiment a bit with their configuration also does.

comment:3 Changed 5 years ago by sa2ajj

  • Version changed from 0.8.9 to master

comment:4 Changed 5 years ago by dustin

I wonder what would be the best interface for that. The options I see are:

  • horizons: delete builders which have no active master and no builds
  • immediate disappearance: if there are no active masters with a builder configured, it disappears
  • command-line cleanup: buildbot delete-builder XXX
  • web UI: administrators can delete builders from the web UI

I'm not sure which of these options is best..

comment:5 Changed 5 years ago by sa2ajj

FWIW, two last options look the most appealing to me (cli & web ui options):

  • people do make mistakes -> give them a chance to see that something is wrong
  • in a multi-master setup it's more likely to make a mistake
Last edited 5 years ago by sa2ajj (previous) (diff)

comment:6 Changed 5 years ago by Ben

During a reconfig, we should be able to flag the dead builders as dead, or not ?

comment:7 Changed 5 years ago by Ben

Same is valid for the slaves by the way. Even worst, the slaves still show on the web interface a connection with the builders they were connected to (when they existed). I.e.: configured_on: [{"builderid":1,"masterid":1},{"builderid":2,"masterid":1},{"builderid":3,"masterid":1}] But that slave is absolutely not configured there ...

Those slave got new names / passwords, hence disapeared, and have been replaced by other ones ...

comment:8 Changed 5 years ago by dustin

Reconfig happens on one master at a time, so it's still tricky at that point. Probably the worst outcome would be to have a builder deleted while its builderid was still cached in RAM in another master -- any new inserts with that ID would fail.

So, yes, I think a way to delete builders (and slaves) manually would be good.

comment:9 Changed 5 years ago by Ben

I believe, we need to move to a next-generation multi-master configuration.

The more we move stuff in the db, the more we are having this kind of troubles which boils down to:

We do have a decentralized system without coordinating entity (well, the db, but it's only passive).

What if we would require for the master.cfg to always contains the whole definition, and then, during buildbot start we would add a parameter to tell the master which one he actually is ?

comment:10 Changed 5 years ago by dustin

What are some good models for this, among other clustered applications?

I think your suggestion is to use the same master.cfg for all masters, with runtime flags that cause it to be interpreted differently. But, how would you enforce that? And how would you accomplish a rolling configuration update?

comment:11 Changed 5 years ago by sa2ajj

  • Keywords multimaster added

comment:12 Changed 5 years ago by dustin

  • Summary changed from Old Builders not defined anymore in the config are still there to Support a single config across the cluster, or something like that
Note: See TracTickets for help on using tickets.