Buildbot won't delete files with Unicode file names

[ipv6guru]

Our app creates log files that may have Chinese characters in the file name. Our tests under buildbot are able to create these files properly the first time buildbot runs them, but any subsequent build attempt fails because buildbot throws a Python exception when it attempts to delete these log files before running the next checkout.

On any build that runs after a build that has created these Chinese-character-named log files, we see exceptions such as the following:

starting svn operation

remoteFailed: [Failure instance: Traceback from remote host -- Traceback
(most recent call last):
File "C:Python24libsite-packages	wistedinternetdefer.py", line 106,
in maybeDeferred
result = f(*args, **kw)
File "C:Python24libsite-packagesuildbotslavecommands.py", line
1216, in start
d.addCallback(self.doClobber, self.workdir)
File "C:Python24libsite-packages	wistedinternetdefer.py", line 195,
in addCallback
callbackKeywords=kw)
File "C:Python24libsite-packages	wistedinternetdefer.py", line 186,
in addCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "C:Python24libsite-packages	wistedinternetdefer.py", line 328,
in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "C:Python24libsite-packagesuildbotslavecommands.py", line
1374, in doClobber
rmdirRecursive(d)
File "C:Python24libsite-packagesuildbotslavecommands.py", line 86,
in rmdirRecursive
rmdirRecursive(full_name)
File "C:Python24libsite-packagesuildbotslavecommands.py", line 86,
in rmdirRecursive
rmdirRecursive(full_name)
File "C:Python24libsite-packagesuildbotslavecommands.py", line 83,
in rmdirRecursive
os.chmod(full_name, 0600)
exceptions.OSError: [Errno 22] Invalid argument:
'D:\buildbot\slaves\ccxengine-fatman\trunk-with-client-acceptance\buil
d\Server\log\FIX.4.4-CCXORDER-????.event.log'
]

Python supports Unicode filenames, but we think perhaps Buildbot is using deprecated API calls.

I see this in 0.7.7. I don't see any existing bugs for this, nor any indication that it is fixed in 0.7.9, but we are checking.

Submitted: Grant Birchmeier ( grantbirchmeier ) - 2009-02-10 02:16

Additional Comments :-

Date: 2009-02-11 05:36 Sender: grantbirchmeier

Hide We are using the following workaround. This code might help in developing a solution. It has only been run on Windows, so investigation is needed to see if this kind of fix is safe for other platforms.

I have modified this function, found in command.py:

def rmdirRecursive(dir):
"""This is a replacement for shutil.rmtree that works better under
windows. Thanks to Bear at the OSAF for the code."""

if not os.path.exists(dir):
return

if os.path.islink(dir):
os.remove(dir)
return

# Verify the directory is read/write/execute for the current user
os.chmod(dir, 0700)

# os.listdir below only returns a list of unicode filenames if the
parameter is unicode
# Thus, if a non-unicode-named dir contains a unicode filename, that
filename will get garbled.
# So force dir to be unicode.
# (This issue is seen on Windows. Not sure how this hack will affect
other platforms. Probably badly.)
udir = ""
try:
udir = unicode(dir, "utf-8")
except TypeError:
udir = dir


for name in os.listdir(udir):
full_name = os.path.join(udir, name)
# on Windows, if we don't have write permission we can't remove
# the file/directory either, so turn that on
if os.name == 'nt':
if not os.access(full_name, os.W_OK):
# I think this is now redundant, but I don't have an NT
# machine to test on, so I'm going to leave it in place
# -warner
os.chmod(full_name, 0600)

if os.path.isdir(full_name):
rmdirRecursive(full_name)
else:
os.chmod(full_name, 0700)
os.remove(full_name)
os.rmdir(dir)

Date: 2009-02-10 17:00 Sender: grantbirchmeier

The problem is present in 0.7.9.

[ipv6guru]

Moved from soureforge

[dustin]

Handling filesystem encodings is always a nightmare. What if the filenames aren't unicode, but are in a Windows charset?

[dustin]

If someone with a windows bot can apply  http://github.com/djmitche/buildbot/commit/bug485.patch and confirm that it works, I'll get this committed.

[dustin]

Never mind that - we have a Windows test bot (duh!).

[Dustin J. Mitchell]

Win32: Try to force dirname to unicode for a better os.remove experience

Fixes #485

Changeset: 6b3e1972ddbcac070a072411417022b4166746ce