HTML logs do not get compressed

[marcusl]

I recently started adding HTML logs from my buildsystem to buildbot, and these are between 1 and 5 mb per buildstep.

They do not seem to be compressed at all, so there is quite a lot of memory and disk-space taken up by these log files. (One of my masters is using 450mb)

Having the logs gzipped would be neat, since they could then be sent raw to clients who accept gzipped html (most browsers nowadays).

Either that, or we fold the HTML-log and text-log into a generic logging store (with mime-type or something).

[marcusl]

Also, the build-pickle seems to contain all logs as well (the 555 file is about the size of the sum of all logs below, similar results for other builds:)

-rw------- 1 root root 10524631 Mar 11 12:57 555
-rw------- 1 root root  1054001 Mar 11 12:26 555-log-build-buildlog
-rw------- 1 root root     9897 Mar 11 12:26 555-log-build-stdio.bz2
-rw------- 1 root root      891 Mar 11 12:26 555-log-build-warnings
-rw------- 1 root root   158168 Mar 11 12:07 555-log-clean-buildlog
-rw------- 1 root root     1915 Mar 11 12:07 555-log-clean-stdio.bz2
-rw------- 1 root root     1797 Mar 11 12:01 555-log-hg-stdio.bz2
-rw------- 1 root root  6237923 Mar 11 12:57 555-log-install-buildlog
-rw------- 1 root root     5587 Mar 11 12:57 555-log-install-stdio.bz2
-rw------- 1 root root       58 Mar 11 12:06 555-log-setup-property_changes
-rw------- 1 root root     5763 Mar 11 12:06 555-log-setup-stdio.bz2
-rw------- 1 root root  3080263 Mar 11 12:35 555-log-test-buildlog
-rw------- 1 root root     7500 Mar 11 12:35 555-log-test-stdio.bz2
-rw------- 1 root root      219 Mar 11 12:35 555-log-test-warnings

I'll try to get some time to test the 0.8 beta and see if it helps, but some initial advice would be nice.

[dustin]

If you can decode one of those pickles and find where the logfile is stuck, that should be easy to fix. I'll be happy to backport that to the 0.8.0 release branch.

Compressing them .. harder, but a good project.

[dustin]

marcus, are you still having this problem? Can you send me a build pickle?

[marcusl]

Uh. Sorry for the quietness. I didn't see your comments.

I'll get back to this once we update our buildbot install to 0.8.x at work. (within a few months..)

[marcusl]

Or rather, on monday, I can try to disassemble the pickle or mail it to you. My vacation for this summer is now over, and I have to work for my salary again. ;)