support binary log files in build steps

[brendan]

I have buildsteps that produce a tarball containing debug information (coredumps etc) when tests fail. I'd like to include these as log files so that I can grab them from the summary page. Adding them as logfiles sort of works, but when I click on the link it comes back as text/html that my browser attempts to render. I'd like to be able to just download it directly. Ideally, I'd just flag a log file as binary in the buildstep constructor.

For reference, I've attached the hack I'm using right now (I am not proposing to include this patch).

[brendan]

shell.Test extended to capture binary logs

[dustin]

That's pretty much the correct strategy. It'd be nice if it was easier, but you seem to have implemented it quite well :)

[brendan]

Well, alright. One of the reasons I don't like my patch is that it relies on an internal API (transfer._FileWriter). It would also be nice to be able to store the file alongside the other build logs in the build directory instead of stuffing them in public_html (for instance, to make it easier to garbage collect them). I think it's fair game for an enhancement request, but if you want to leave it closed I'm not going to keep reopening it :)

[brendan]

(reopening just this once, in case comments just go into the bitbucket otherwise)

[dustin]

There's a larger question in buildbot of how to deal with build artifacts. I think you'll agree that these are *not* logs, and don't warrant the same treatment. The file upload/download support is fairly primitive, and the ways to expand this would be:

• track uploaded files explicitly in the build status, so that they can be expired with something similar to the logHorizon, and are automatically linked from the build/step that generated them
• allow files to flow through some other channel than the master/slave channel, as this channel - being RPC-based - is less than optimal for transfer of large files, and also places a lot of load on the master.

I'm going to re-close this bug, as you've addressed the particular purpose of your request, but if you're interested in doing some more general work on artifact handling, please get in touch on the mailing list!