Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#1752 closed undecided (fixed)

xp slave spews rmdirRecursive messages during svn update

Reported by: philippem Owned by:
Priority: minor Milestone: undecided
Version: 0.8.3 Keywords:
Cc:

Description

  • buildbot slave on xp sp3, buildbot 0.8.3, python 2.7, Twisted 10.2.0
  • during step.SVN update on the slave, the buildbot.tac file is filed with hundreds of messages like:

[2011-01-04 13:03:18-0800 [broker,client] rmdirRecursive: decoding from UTF-8 failed

The build however continues without problems.

Change History (10)

comment:1 Changed 9 years ago by philippem

my mistake, the twistd.log file is filled with the messages, not the buildbot.tac. The svn update step in the master.cfg is:

f2.addStep(source.SVN(mode='update', baseURL='svn://builder/XXXX/', defaultBranch='trunk', always_purge=False))

comment:2 Changed 9 years ago by dustin

What sorts of files is it deleting? There's some hacky code to handle encoded filenames:

https://github.com/buildbot/buildbot/blob/master/slave/buildslave/commands/utils.py#L59

that's probably giving you this grief.

comment:3 Changed 9 years ago by philippem

grief is a strong word. :-) it's deleting obj and dll files from the previous compile. I haven't seen non ascii file names, but I've added some more logging to this exception.

BTW is there a place I can look, besides the buildbot source, for examples of how to create build steps? A cookbook of sorts, of examples, would be wonderful.

thanks!

comment:4 Changed 9 years ago by dustin

At this point, no, there's nowhere to look for that, and in fact it's not even well-specified. Fixing that is the plan for Buildbot-0.10.+.

comment:5 Changed 9 years ago by philippem

thanks for the quick replies and support, it's much appreciated. buildbot has been great for me, and it's a wonderful system. It's well engineered and its features obviously come from experience and concrete motivations.

One of the challenges I've had with it, and not being a python wizard makes it tough, is doing things slightly outside the provided classes. It then becomes necessary to create derived build steps, and (this may be true of any OO framework), it takes a lot of time to figure out what the assumptions are and what is happening when, because there are factories, logging, process control, events on remote machines, etc. This is usually done by grepping through the buildbot source.

For example, I have a Test step that generates an html summary on the slave when it completes. I want this html summary to appear as an (html) log file in the waterfall, and I want to supress a text version of this log file. I've experimented with various approaches (including attempting a remote shell command as in the python_twisted source), but finally using addHTMLLog (is there a removeLog?). What would be some approaches to solving this? (StatusPlugin??)

thanks!

comment:6 Changed 9 years ago by dustin

Yes, I don't deny it's a problem and makes it hard to use Buildbot deeply. It's a long-term project, though, because many of the interfaces are not well-defined, and because we need to proceed in a way that matches existing assumptions people have made as closely as possible, without breaking abstractions that Buildbot needs to grow.

comment:7 Changed 9 years ago by philippem

Here are some sample messages, I inserted some debug code to print the dirs. It's not just .svn meta dirs that cause the message however.

2011-01-10 20:25:35-0800 [-] rmdirRecursive: decoding of c:Source3rdpartysrce
ncodersflac-1.2.1srclibFLACinclude.svn	mpprop-base from UTF-8 failed exce
ption=decoding Unicode is not supported
2011-01-10 20:25:35-0800 [-] rmdirRecursive: decoding of c:Source3rdpartysrce
ncodersflac-1.2.1srclibFLACinclude.svn	mpprops from UTF-8 failed exceptio
n=decoding Unicode is not supported
2011-01-10 20:25:35-0800 [-] rmdirRecursive: decoding of c:Source3rdpartysrce
ncodersflac-1.2.1srclibFLACinclude.svn	mp	ext-base from UTF-8 failed exce
ption=decoding Unicode is not supported

comment:8 Changed 9 years ago by Dustin J. Mitchell

  • Resolution set to fixed
  • Status changed from new to closed

Don't try to re-decode a Unicode string from UTF-8

Python doesn't support trying to interpret a Unicode string as utf-8 - you should be decoding a *bytestring* into utf-8 instead. So don't do that. Fixes #1752.

Changeset: affdeceb68cd314e430add18c89f74fc394d18f0

comment:9 Changed 9 years ago by dustin

philippem: can you try applying https://github.com/buildbot/buildbot/commit/affdeceb68cd314e430add18c89f74fc394d18f0.patch and let me know if it helps? Please reopen if it doesn't fix the problem.

comment:10 Changed 9 years ago by philippem

patch applied, thank you. I will see what happens.

Note: See TracTickets for help on using tickets.