Changes sent over the wire should be compressed

[dank]

There is currently an arbitrary 640KB limit on the size of changes, imposed by twisted.spread.banana.SIZE_LIMIT, see  http://twistedmatrix.com/documents/current/core/howto/pb-limits.html

This becomes a problem when patches are sent that replace large components (see e.g. the nine patch series at  http://www.winehq.org/pipermail/wine-patches/2011-September/107181.html, which, when concatenated to be submitted as a try, hits about 900KB. )

Either that limit needs to be raised or removed, or changes should be compressed, or both. On slow links, compression would be a big win.

[dustin]

So there are two things to do in this bug:

• make twisted.spread.banana.SIZE_LIMIT configurable
• compress patches

The first is fairly straightforward, and with a little care in protocol design, could even be configured only on the master. The second may occur automatically when support for patches is added to the master-side source steps.

[dustin]

In fact, once patches are transferred using FileDownload, there won't be any need to change SIZE_LIMIT, so maybe the first point is just a documentation change to suggest adding

from twisted.spread import banana
banana.SIZE_LIMIT = 1024**2

to buildbot.tac on the master and slave (with suitable security warnings).

[tom.prince]

Using buildbot try with the PBListner will also trigger this.

[dustin]

Indeed - this is a fundamental flaw of naïve use of RPC for data transfer, in both cases.

Compressing patches will only buy time (space?) until this hurts a particular use. At least changing the SIZE_LIMIT lets the user dial in their own ceiling.

Let's do the documentation as a simple step to get 0.8.6 out the door. If there's clamor for it, we can later implement this as a master configuration parameter (and call a method on the slave to set the corresponding value, and similarly get the try client to determine the appropriate SIZE_LIMIT before sending a patch).

[dank]

Documenting the limit and how to tweak it would be helpful (especially if you emit a warning pointing to the doc when the limit is exceeded).

Later, though, compression would be a big win in some cases.

[dustin]

Hm, this may need to be adjusted in the buildbot try client, too - dank, can you verify how best to set SIZE_LIMIT in that case? I was thinking of the master/slave communication, which is just a buildbot.tac modification.

[dank]

Both my client and my server install scripts contain this line:

    # Large patches cause problems without this.  Need on both sides,
    # see http://twistedmatrix.com/documents/current/core/howto/pb-limits.html
    sed -i.bak -e 's/SIZE_LIMIT =.*/SIZE_LIMIT = 2 * 1024 * 1024/' 
        $TOP/sandbox/lib/python2.?/site-packages/twisted/spread/banana.py

[dustin]

This is related both to replacing try with force (#896) and a new master/slave protocol (#2437). Once both of those are fixed, this will be fixed. Until then, the fix here is good enough.

[dank]

To be picky, the command I posted is a workaround, not a fix.