sqlite3.ProgrammingError when starting a rebuild with non-ASCII chars in re-build reason

[andialbrecht]

I've hit this bug when I tried to start a re-build of a failed build and entered a message with non-ASCII chars in the "Reason for re-build" textbox (buildbot 0.8.1p1 with SQLite database):

/home/build/.virtualenvs/buildbot/lib/python2.6/site-packages/buildbot/db/dbspec.py, line 105 in execute
103      try:
104        query_start_time = util.now()
105        result = self.cursor.execute(*args, **kw)
106        end_time = util.now()
Self
cursor	<sqlite3.Cursor object at 0xa8fda70>
Locals
kw	Dictionary instance @ 0xa96f604
query_start_time	1289488174.3802421
self	
buildbot.db.dbspec.RetryingCursor instance @ 0xa9f02ec
<sqlite3.Cursor object at 0xa8fda70>
args	('INSERT INTO buildsets (external_idstring, reason, sourcestampid, submitted_at) VALUES (?,?,?,?)', (None, "The web-page 'rebuild' button was pressed by '<unknown>': Tabellenspalte ergxc3xa4nzt
", 186, 1289488174.3802321))
Globals
util	<module 'buildbot.util' from '/home/build/.virtualenvs/buildbot/lib/python2.6/site-packages/buildbot/util/__init__.pyc'>
<class 'sqlite3.ProgrammingError'>: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

[dustin]

Sqlalchemy should clean this up. Leaving this open to check that assertion when SQLAlchemy is in place.

[dustin]

I still get this when rebuilding because "☃"

<class 'sqlalchemy.exc.ProgrammingError?'>: (ProgrammingError?) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'INSERT INTO buildsets (external_idstring, reason, submitted_at, complete, complete_at, results, sourcestampsetid) VALUES (?, ?, ?, ?, ?, ?, ?)' (None, "The web-page 'rebuild' button was pressed by '<unknown>': xe2x98x83 ", 1331509675.957143, 0, None, -1, 2)

so this definitely isn't fixed.

[dustin]

Ah, so the issue here is that the web status is feeding us bytestrings, and we're not decoding them properly.

According to  http://stackoverflow.com/questions/708915/detecting-the-character-encoding-of-an-http-post-request it's not quite right to assume utf-8 here. Sadly.

[Dustin J. Mitchell]

Decode form values from HTTP POSTs

Fixes #1054. One hopes.

Changeset: c8079379bc130997e97ab7e4b96aebaae1bf3912

[Dustin J. Mitchell]

Decode form values from HTTP POSTs

Fixes #1054. One hopes.

Changeset: c8079379bc130997e97ab7e4b96aebaae1bf3912

[Tom Prince]

Also decode the filed names when force-building.

With ForceScheduler?, it is possible to have semi-arbitrary names. This also cleans up the arg-preprocessing code somewhat.

Refs #1054.

Changeset: 92231c0926c6d7725f98cfdeb2201088badb1ce8

[Tom Prince]

Also decode the filed names when force-building.

With ForceScheduler?, it is possible to have semi-arbitrary names. This also cleans up the arg-preprocessing code somewhat.

Refs #1054.

Changeset: 92231c0926c6d7725f98cfdeb2201088badb1ce8