PBChangeSource loses string encodings

[dustin]

If you run 'buildbot sendchange' and pass utf-8 strings on the command line, they will arrive as bytestrings on the PBChangeSource, and great sadness will ensue:

2011-02-22 13:46:58+0000 [Broker,5,172.31.17.193] perspective_addChange called
2011-02-22 13:46:58+0000 [Broker,5,172.31.17.193] Peer will receive following PB
 traceback:
2011-02-22 13:46:58+0000 [Broker,5,172.31.17.193] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.6/dist-packages/twisted/spread/banana.py", line
 153, in gotItem
            self.callExpressionReceived(item)
          File "/usr/lib/python2.6/dist-packages/twisted/spread/banana.py", line
 116, in callExpressionReceived
            self.expressionReceived(obj)
          File "/usr/lib/python2.6/dist-packages/twisted/spread/pb.py", line 514
, in expressionReceived
            method(*sexp[1:])
          File "/usr/lib/python2.6/dist-packages/twisted/spread/pb.py", line 826
, in proto_message
            self._recvMessage(self.localObjectForID, requestID, objectID, messag
e, answerRequired, netArgs, netKw)
        --- <exception caught here> ---
          File "/usr/lib/python2.6/dist-packages/twisted/spread/pb.py", line 840
, in _recvMessage
            netResult = object.remoteMessageReceived(self, message, netArgs, net
Kw)
          File "/usr/lib/python2.6/dist-packages/twisted/spread/pb.py", line 225
, in perspectiveMessageReceived
            state = method(*args, **kw)
          File "/usr/local/lib/python2.6/dist-packages/buildbot/changes/pb.py", 
line 58, in perspective_addChange
            self.changemaster.addChange(change)
          File "/usr/local/lib/python2.6/dist-packages/buildbot/changes/manager.
py", line 82, in addChange
            change.comments, change.category, change.project))
        exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in p
osition 27: ordinal not in range(128)
        

[dustin]

I suspect that the right solution here is to allow the sendchange user to specify an encoding for the command line (defaulting to the Python filesystem encoding?), and decode from that encoding to unicode strings before sending to PBChangeSource.

I don't recall how well PB handles unicode, so that should be tested in a unit test that will detect any failures in older versions.

PBChangeSource should be resilient by trying to decode any raw bytestrings it receives as ascii with replacement.

[dustin]

Twistd-8.2.0's PB handles unicode just fine in my manual testing.

[Dustin J. Mitchell]

Handle bytestrings on input to PBChangeSource

This handles bytestrings by assuming, softly, that they are in utf8. Ideally the PB client would not send bytestrings, but at least this avoids crashing. Refs #1829.

Changeset: dcc08973694d6c6dc2f482c83bd51317c2839c07

[Dustin J. Mitchell]

Try to decode strings in 'buildbot sendchange'

By default, this uses utf8, but that can be specified on the command line with --encoding. Refs #1829.

Changeset: 605ac9aaf644613a43f076fd25ddcbef93a6c087