Opened 9 years ago

Closed 6 years ago

#951 closed defect (fixed)

UTF-8 data in logs causes web rss feed exception

Reported by: AHowell Owned by: AHowell
Priority: major Milestone: 0.8.2
Version: 0.8.1 Keywords: web
Cc:

Description (last modified by dustin)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/twisted/protocols/basic.py", line 251, in dataReceived

why = self.lineReceived(line)

File "/usr/lib/python2.7/site-packages/twisted/web/http.py", line 1573, in lineReceived

self.allContentReceived()

File "/usr/lib/python2.7/site-packages/twisted/web/http.py", line 1641, in allContentReceived

req.requestReceived(command, path, version)

File "/usr/lib/python2.7/site-packages/twisted/web/http.py", line 807, in requestReceived

self.process()

--- <exception caught here> ---

File "/usr/lib/python2.7/site-packages/twisted/web/server.py", line 125, in process

self.render(resrc)

File "/usr/lib/python2.7/site-packages/twisted/web/server.py", line 132, in render

body = resrc.render(self)

File "/usr/lib/python2.7/site-packages/buildbot/status/web/feeds.py", line 40, in render

data = self.content(request)

File "/usr/lib/python2.7/site-packages/buildbot/status/web/feeds.py", line 202, in content

log_lines.extend(unicode(logdata.split('

')[-30:],'utf-8'))

exceptions.TypeError?: coercing to Unicode: need string or buffer, list found

Change History (6)

comment:1 Changed 9 years ago by AHowell

  • Resolution set to fixed
  • Status changed from new to closed

This fixes it. Is there a better way to create unicode transformed data. My python abilities are limited...

--- a/master/buildbot/status/web/feeds.py +++ b/master/buildbot/status/web/feeds.py @@ -198,8 +198,10 @@ class FeedResource?(XmlResource?):

except IOError:

# Probably the log file has been removed logdata =' log file not available '

-

  • log_lines.extend(logdata.split('

')[-30:]) + unilist = list() + for line in logdata.split(' ')[-30:]: + unilist.append(unicode(line,'utf-8')) + log_lines.extend(unilist)

bc = {} bcdate? = time.strftime("%a, %d %b %Y %H:%M:%S GMT",

comment:2 Changed 9 years ago by dustin

  • Keywords web added
  • Milestone changed from undecided to 0.8.2
  • Resolution fixed deleted
  • Status changed from closed to reopened

Not fixed yet :)

comment:3 Changed 9 years ago by AndyHowell

  • Resolution set to fixed
  • Status changed from reopened to closed

Fixes #951. Web rss feed is choking on UTF-8 log data

Changeset: 5cbdcd768dc9e2297181b301e11ddae5f767e4a2

comment:4 Changed 9 years ago by Dustin J. Mitchell

Merge branch 'cvsmailsource' of git://github.com/AndyHowell/buildbot

Fixes #959

  • 'cvsmailsource' of git://github.com/AndyHowell/buildbot: option.smtplib was not getting when running live Added depreciation notes to news and manual Added more test to test_changes_mail_CVSMaildirSource.py and test_contrib_buildbot_cvs_mail.py Fixed mail.py problems brought out by tests use optparse for buildbot_cvs_mail; fix its tests rename to CVSMaildirSource fix relative path handling in buildbot_cvs_mail tests docs tweaks for cvs support buildbot/changes/mail.py buildbot/test/unit/test_changes_mail_MaildirSource.py removed fileTupleList as it was not being used master/buildbot/changes/mail.py master/contrib/buildbot_cvs_mail.py Replaced tabs with spaces master/buildbot/changes/mail.py added BuildbotCVSMaildir source master/buildbot/test/unit/test_changes_mail_MaildirSource.py test for BuildbotCVSMaildir in mail.py master/buildbot/test/unit/test_contrib_buildbot_cvs_mail.py test for buildbot_cvs_mail.py master/contrib/buildbot_cvs_mail.py send mail to buildbot on CVS checkins master/docs/cfg-changesources.texinfo updated docs Fixes #951. Web rss feed is choking on UTF-8 log data Changeset: b9feff0338ecd7cceead5ede70852abc1f0a6d2a

comment:5 Changed 6 years ago by paolog

  • Resolution fixed deleted
  • Status changed from closed to reopened

Not fixed yet.

I have this text in one of the logs: ""tar" non � riconosciuto come comando interno o esterno, un programma eseguibile o un file batch." - this was generated on an Windows 8 buildslave with Italian language settings.

When I extract the feed with http://localhost:8080/atom?failures_only=true an a Debian 7 buildmaster (that' 0.8.6.) it returns:

web.Server Traceback (most recent call last):

<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte 0x8a in position 10: invalid start byte
/usr/lib/python2.7/dist-packages/twisted/web/server.py, line 132 in process
130    try:
131      resrc = self.site.getResourceFor(self)
132      self.render(resrc)
133    except:
/usr/lib/python2.7/dist-packages/twisted/web/server.py, line 167 in render
165    """
166    try:
167      body = resrc.render(self)
168    except UnsupportedMethod, e:
/usr/lib/python2.7/dist-packages/buildbot/status/web/feeds.py, line 55 in render
53
54  def render(self, request):
55    data = self.content(request)
56    request.setHeader("content-type", self.contentType)
/usr/lib/python2.7/dist-packages/buildbot/status/web/feeds.py, line 225 in content
223            for line in logdata.split('\n')[-30:]:
224              # unilist.append(unicode(line,'utf-8','replace'))
225              unilist.append(unicode(line,'utf-8'))
226            log_lines.extend(unilist)

<type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte 0x8a in position 10: invalid start byte

The following fix works: in /usr/lib/python2.7/dist-packages/buildbot/status/web/feeds.py at line 224 change

unilist.append(unicode(line,'utf-8'))

to

unilist.append(unicode(line,'utf-8','replace'))

see http://docs.python.org/2/library/functions.html?highlight=unicode#unicode

comment:6 Changed 6 years ago by dustin

  • Description modified (diff)
  • Resolution set to fixed
  • Status changed from reopened to closed

There's a much better fix in nine (logs are stored in unicode), but this fix is fine for 0.8.x.

commit fdf1f2fdd731403ed96517777429e59531c41a37
Author: Dustin J. Mitchell <dustin@mozilla.com>
Date:   Wed Jan 29 10:23:42 2014 -0500

    Ignore decoding errors in atom/rss feeds; fixes #951
Note: See TracTickets for help on using tickets.