Opened 3 years ago

Closed 19 months ago

Last modified 19 months ago

#2123 closed enhancement (worksforme)

Changes sent over the wire should be compressed

Reported by: dank Owned by:
Priority: minor Milestone: 0.8.+
Version: 0.8.5 Keywords:
Cc:

Description

There is currently an arbitrary 640KB limit on the size of changes, imposed by twisted.spread.banana.SIZE_LIMIT, see http://twistedmatrix.com/documents/current/core/howto/pb-limits.html

This becomes a problem when patches are sent that replace large components (see e.g. the nine patch series at http://www.winehq.org/pipermail/wine-patches/2011-September/107181.html, which, when concatenated to be submitted as a try, hits about 900KB. )

Either that limit needs to be raised or removed, or changes should be compressed, or both. On slow links, compression would be a big win.

Change History (10)

comment:1 Changed 3 years ago by dustin

  • Milestone changed from undecided to 0.8.6

So there are two things to do in this bug:

  • make twisted.spread.banana.SIZE_LIMIT configurable
  • compress patches

The first is fairly straightforward, and with a little care in protocol design, could even be configured only on the master. The second may occur automatically when support for patches is added to the master-side source steps.

comment:2 Changed 3 years ago by dustin

In fact, once patches are transferred using FileDownload, there won't be any need to change SIZE_LIMIT, so maybe the first point is just a documentation change to suggest adding

from twisted.spread import banana
banana.SIZE_LIMIT = 1024**2

to buildbot.tac on the master and slave (with suitable security warnings).

comment:3 Changed 3 years ago by tom.prince

Using buildbot try with the PBListner will also trigger this.

comment:4 Changed 3 years ago by dustin

Indeed - this is a fundamental flaw of naïve use of RPC for data transfer, in both cases.

Compressing patches will only buy time (space?) until this hurts a particular use. At least changing the SIZE_LIMIT lets the user dial in their own ceiling.

Let's do the documentation as a simple step to get 0.8.6 out the door. If there's clamor for it, we can later implement this as a master configuration parameter (and call a method on the slave to set the corresponding value, and similarly get the try client to determine the appropriate SIZE_LIMIT before sending a patch).

comment:5 Changed 3 years ago by dank

Documenting the limit and how to tweak it would be helpful (especially if you emit a warning pointing to the doc when the limit is exceeded).

Later, though, compression would be a big win in some cases.

comment:6 Changed 3 years ago by dustin

Hm, this may need to be adjusted in the buildbot try client, too - dank, can you verify how best to set SIZE_LIMIT in that case? I was thinking of the master/slave communication, which is just a buildbot.tac modification.

comment:7 Changed 3 years ago by dank

Both my client and my server install scripts contain this line:

    # Large patches cause problems without this.  Need on both sides,
    # see http://twistedmatrix.com/documents/current/core/howto/pb-limits.html
    sed -i.bak -e 's/SIZE_LIMIT =.*/SIZE_LIMIT = 2 * 1024 * 1024/' 
        $TOP/sandbox/lib/python2.?/site-packages/twisted/spread/banana.py

Last edited 3 years ago by dank (previous) (diff)

comment:8 Changed 3 years ago by tom.prince

  • Milestone changed from 0.8.6 to 0.8.+

comment:9 Changed 19 months ago by dustin

  • Resolution set to worksforme
  • Status changed from new to closed

This is related both to replacing try with force (#896) and a new master/slave protocol (#2437). Once both of those are fixed, this will be fixed. Until then, the fix here is good enough.

comment:10 Changed 19 months ago by dank

To be picky, the command I posted is a workaround, not a fix.

Note: See TracTickets for help on using tickets.