wiki:MasterSlaveCommunication

There is a new message-based protocol under development. This will replace the Twisted Perspective-Broker based RPC protocol.

Current Protocol

For the current API, See: Master-Worker API

The current protocol sequence diagram, if represented as messages, looks similar to this:

SequenceDiagram

Most remote calls are asynchronous.

Initial Interaction:

Authentication: Currently the worker authenticates with the Twisted Cred (newcred) mechanism, using the username and password supplied to buildslave create-slave.

This occurs as part of the setup of the PB connection, and so doesn't fit into the RPC style of the rest of the current protocol. As it happens, the login mechanism is also used to allow the same port to be used for other PB communications.

After that the master requests information about the worker, via the remote methods getSlaveInfo, getVersion, and getCommands.

The SlaveInfo currently consists of the contents of the worker's info/ directory.
This also contains the keys
environ: copy of the workers environment
system: OS the worker is running (extracted from pythons os.name)
basedir:base directory where worker is running

getVersion returns the version of worker. getCommands returns the list of (name, version) for all commands the worker recognizes. These methods do not take any arguments.

Next the master sets the builder list via setBuilderList. This method, given a list of builders and their build directories, ensures that those builders, and only those builders, are running. It returns a set of SlaveBuilder objects. This method can be called later as well.

Then the master sets itself as master for each SlaveBuilder via setMaster. This returns a reference to the master-side SlaveBuilder.

Reactor Loop:

The twisted event loop, or reactor, then waits for external triggers, such as Build.

On receiving the Build trigger, the master then runs commands on the SlaveBuilder. For documentation on commands see : Remote Commands
For each of the commands,
The command is started with startCommand. This takes stepref, stepId, command, args as arguments.
The command can be interrupted via an external trigger, which calls interruptCommand. This method takes stepID and a reason why as arguments.

The running command provides Updates, sent as arguments to update from the SlaveBuilder to the master, during execution.

At the end of a command, the completion of the command is indicated by calling complete, with failures as optional arguments.

The master and worker (in versions greater than 0.8.3) can initiate the shutdown of a worker. The master can call shutdown to shut down the worker, while the worker can also request for a graceful shutdown.

Which it does by calling shutdown on the master.

Proposed Protocol

The proposed protocol is message-based rather than RPC based. The protocol would be versioned. The initially planned changes are shown below.

ProposedSequenceDiagram

Initial changes will include reducing the number of roundtrips, a busy-signal from the worker, protocol versioning, graceful handling of disconnections, etc. Future changes would include support for several masters, etc.

Last modified 10 months ago Last modified on Jan 4, 2017, 1:39:57 AM

Attachments (2)

Download all attachments as: .zip