Opened 4 years ago

Closed 4 years ago

#3391 closed undecided (wontfix)

UnicodeDecodeError in the logobserver when encountering non-ascii characters

Reported by: Ben Owned by: dustin
Priority: major Milestone: 0.8.x
Version: 0.8.12 Keywords:
Cc:

Description

Traceback (most recent call last):
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1171, in _inlineCallbacks
    deferred.callback(e.value)
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 393, in callback
    self._startRunCallbacks(result)
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/process/buildstep.py", line 650, in <lambda>
    d.addCallback(lambda res: self.commandComplete(cmd))
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/steps/shell.py", line 337, in commandComplete
    self.observer.getStdout(),
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/process/logobserver.py", line 121, in getStdout
    return self._get(self.stdout)
  File "/home/buildsystem/bb-master/local/lib/python2.7/site-packages/buildbot/process/logobserver.py", line 118, in _get
    return u''.join(chunks)
exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 109: ordinal not in range(128)

The Shell step just does a cat of a file that contains non-ascii characters.

Change History (3)

comment:1 Changed 4 years ago by Ben

Just replaced data with data.decode('utf-8') to outReceived and errReceived and the error is gone.

This is probably not the best way to fix this though.

comment:2 Changed 4 years ago by dustin

  • Owner set to dustin
  • Status changed from new to assigned

I'll take a look at this -- in nine, and I think in new-style steps, logs have to be unicode. I think.

comment:3 Changed 4 years ago by dustin

  • Resolution set to wontfix
  • Status changed from assigned to closed

In nine, if your step does not produce utf-8 output, you can set logEncoding on the step to properly decode to unicode. That's not supported in eight, though. In eight, we just assume that all logs are utf-8. This assumption has not been enforced, though, and it is quite possible that things mostly worked for non-utf-8 output until now. Encodings were all over the place in eight.

We have an integration test to check this situation in nine.

I suspect that what you've done is the best fix for eight.

Note: See TracTickets for help on using tickets.