MailNotifier chokes on try job with non-ascii char in first line of patch?

[dank]

I'm turning MailNotifier? on, and the very first job it tried to process happened to be a try job where the first line of the patch was "From: André..." (with the non-ascii character at offset 10 in the patch). This did not go over well:

File "buildbot/status/builder.py", line 535, in _buildFinished

w.buildFinished(name, s, results)

File "buildbot/status/mail.py", line 411, in buildFinished

return self.buildMessage(name, [build], results)

File "buildbot/status/mail.py", line 596, in buildMessage

results, builds, patches, logs)

File "buildbot/status/mail.py", line 526, in createEmail

a = MIMEText(patch[1].encode(ENCODING), _charset=ENCODING)

exceptions.UnicodeDecodeError?: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)

Evidently it thinks the patch is ascii, and explodes when trying to convert it to utf-8.

This is with buildbot from git as of yesterday, 077078052a1fe259d31b081d9df60ff5e15fec01

The try job was submitted with buildbot try --who nerv@… --properties=comment="78111: Subject: wineview: Rename view to wineview and install it" --connect=pb --master=127.0.0.1:5555 --username=blah --passwd=blah --diff=78111.patch -p

and 78111.patch was fetched from  http://source.winehq.org/patches/data/78111 using the command wget -O 78111.patch  http://source.winehq.org/patches/data/78111

[dustin]

Hmm, patches don't generally come with an encoding - they're bytestrings. I suspect that the best we can do here is to try to decode the patch using a default encoding, ignoring errors, and then re-encode it as utf8.

[Dustin J. Mitchell]

Handle non-unicode patches in MailNotifier?

This makes the reasonable assumption that most patches will be interpretable as unicode, even though they are fundamentally bytestrings. Where such interpretation fails, they will be attached to the mail as binary blobs. Fixes #2091.

Changeset: 6c0f1bb630f42df4cb5b02de3e4c2349e196dca8

[dank]

Still failing. Here's the current stack trace with yesterday's git:

2011-09-18 11:33:40-0700 [-] Exception caught notifying <buildbot.status.mail.MailNotifier instance at 0xa00212c> of buildFinished event
2011-09-18 11:33:40-0700 [-] Unhandled Error
  Traceback (most recent call last):
    File "buildbot-git/master/buildbot/status/build.py", line 290, in buildFinished
      w.callback(self)
    File "lib/python2.7/site-packages/twisted/internet/defer.py", line 361, in callback
      self._startRunCallbacks(result)
    File "lib/python2.7/site-packages/twisted/internet/defer.py", line 455, in _startRunCallbacks
      self._runCallbacks()
    File "lib/python2.7/site-packages/twisted/internet/defer.py", line 542, in _runCallbacks
      current.result = callback(current.result, *args, **kw) 
  --- <exception caught here> ---
    File "buildbot-git/master/buildbot/status/builder.py", line 535, in _buildFinished
      w.buildFinished(name, s, results)
    File "buildbot-git/master/buildbot/status/mail.py", line 412, in buildFinished
      return self.buildMessage(name, [build], results)
    File "buildbot-git/master/buildbot/status/mail.py", line 618, in buildMessage
      results, builds, patches, logs) 
    File "buildbot-git/master/buildbot/status/mail.py", line 520, in createEmail
      text = msgdict['body'].encode(ENCODING)
  exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 582: ordinal not in range(128)

Not sure what it was unhappy about, the patch it was trying to attach is  http://source.winehq.org/patches/data/79014 From: Francois Gouget <fgouget@…> Subject: msxml3: Fix printing NULL strings. and has no funny characters in it anywhere I can see.

[dustin]

I think we'll need to defer this to the next release..

[dank]

Hah. It happened again, and I took a closer look. Note the tail end of the exception:

File "buildbot-git/master/buildbot/status/mail.py", line 520, in createEmail

text = msgdictbody?.encode(ENCODING)

exceptions.UnicodeDecodeError?: 'ascii' codec can't decode byte 0xe2 in position 582: ordinal not in range(128)

This is *before* attachments happen; it's in the body. Suspecting it was in the summary of the error that my custom message formatter inserts, I looked for the byte it couldn't decode:

wget  http://buildbot.kegel.com:8010/builders/runtests-default-x86_64/builds/55/steps/shell_2/logs/stdio/text hexdump -C text > hex grep -C3 'e2 ' hex

And voila:

0014f9c0 2d 75 6e 75 73 65 64 2d 62 75 74 2d 73 65 74 2dunused-but-set-|

0014f9d0 76 61 72 69 61 62 6c 65 22 0a 74 6d 61 72 73 68 |variable".tmarsh| 0014f9e0 61 6c 2e 63 3a 20 49 6e 20 66 75 6e 63 74 69 6f |al.c: In functio| 0014f9f0 6e 20 e2 80 98 73 65 72 69 61 6c 69 7a 65 5f 70 |n ...serialize_p| 0014fa00 61 72 61 6d e2 80 99 3a 0a 74 6d 61 72 73 68 61 |aram...:.tmarsha| 0014fa10 6c 2e 63 3a 37 31 37 3a 32 37 39 3a 20 77 61 72 |l.c:717:279: war| 0014fa20 6e 69 6e 67 3a 20 63 61 73 74 20 74 6f 20 70 6f |ning: cast to po| 0014fa30 69 6e 74 65 72 20 66 72 6f 6d 20 69 6e 74 65 67 |inter from integ|

gcc itself is using non-ascii characters in its error messages! The line generated.c: In function ‘test_pack_LPNONCLIENTMETRICSA’: uses a non-ascii opening single quote.

So this newer crash is different from the older one, and we can close this bug.

(Easy workaround for me: when running gcc, use LANG=C. Alternately, figure out how to do a destructive mapping to ascii in my custom message formatter.)

[dustin]

Hah, this "feature" of GCC has annoyed me many times!

You can encode to ascii with str.encode('ascii', 'replace'), by the way.

[dank]

For completeness, I should mention that I had to set LC_CTYPE=C, too, else gcc would continue using the funky character. (Probably because buildbot sets LC_CTYPE explicitly, so it needs overriding.)