Ticket #485 (closed defect: fixed)
Buildbot won't delete files with Unicode file names
|Reported by:||ipv6guru||Owned by:|
Description (last modified by dustin) (diff)
Our app creates log files that may have Chinese characters in the file name. Our tests under buildbot are able to create these files properly the first time buildbot runs them, but any subsequent build attempt fails because buildbot throws a Python exception when it attempts to delete these log files before running the next checkout.
On any build that runs after a build that has created these Chinese-character-named log files, we see exceptions such as the following:
starting svn operation
remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): File "C:Python24libsite-packages wistedinternetdefer.py", line 106, in maybeDeferred result = f(*args, **kw) File "C:Python24libsite-packagesuildbotslavecommands.py", line 1216, in start d.addCallback(self.doClobber, self.workdir) File "C:Python24libsite-packages wistedinternetdefer.py", line 195, in addCallback callbackKeywords=kw) File "C:Python24libsite-packages wistedinternetdefer.py", line 186, in addCallbacks self._runCallbacks() --- <exception caught here> --- File "C:Python24libsite-packages wistedinternetdefer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "C:Python24libsite-packagesuildbotslavecommands.py", line 1374, in doClobber rmdirRecursive(d) File "C:Python24libsite-packagesuildbotslavecommands.py", line 86, in rmdirRecursive rmdirRecursive(full_name) File "C:Python24libsite-packagesuildbotslavecommands.py", line 86, in rmdirRecursive rmdirRecursive(full_name) File "C:Python24libsite-packagesuildbotslavecommands.py", line 83, in rmdirRecursive os.chmod(full_name, 0600) exceptions.OSError: [Errno 22] Invalid argument: 'D:\buildbot\slaves\ccxengine-fatman\trunk-with-client-acceptance\buil d\Server\log\FIX.4.4-CCXORDER-????.event.log' ]
Python supports Unicode filenames, but we think perhaps Buildbot is using deprecated API calls.
I see this in 0.7.7. I don't see any existing bugs for this, nor any indication that it is fixed in 0.7.9, but we are checking.
Submitted: Grant Birchmeier ( grantbirchmeier ) - 2009-02-10 02:16
Additional Comments :-
Date: 2009-02-11 05:36 Sender: grantbirchmeier
Hide We are using the following workaround. This code might help in developing a solution. It has only been run on Windows, so investigation is needed to see if this kind of fix is safe for other platforms.
I have modified this function, found in command.py:
def rmdirRecursive(dir): """This is a replacement for shutil.rmtree that works better under windows. Thanks to Bear at the OSAF for the code.""" if not os.path.exists(dir): return if os.path.islink(dir): os.remove(dir) return # Verify the directory is read/write/execute for the current user os.chmod(dir, 0700) # os.listdir below only returns a list of unicode filenames if the parameter is unicode # Thus, if a non-unicode-named dir contains a unicode filename, that filename will get garbled. # So force dir to be unicode. # (This issue is seen on Windows. Not sure how this hack will affect other platforms. Probably badly.) udir = "" try: udir = unicode(dir, "utf-8") except TypeError: udir = dir for name in os.listdir(udir): full_name = os.path.join(udir, name) # on Windows, if we don't have write permission we can't remove # the file/directory either, so turn that on if os.name == 'nt': if not os.access(full_name, os.W_OK): # I think this is now redundant, but I don't have an NT # machine to test on, so I'm going to leave it in place # -warner os.chmod(full_name, 0600) if os.path.isdir(full_name): rmdirRecursive(full_name) else: os.chmod(full_name, 0700) os.remove(full_name) os.rmdir(dir)
Date: 2009-02-10 17:00 Sender: grantbirchmeier
The problem is present in 0.7.9.
- Keywords encoding added; sourceforge 2582152 removed
- Milestone changed from 0.8.+ to 0.8.1