Please add the ability to automatically delete unused build directories

[smat]

My team runs a large buildbot install (currently 143 builders across 4 platforms), with frequent addition and removal of builds. We find that accumulation of unused build directories on the slaves creates disk usage problems. It would be excellent if Buildbot were to automatically delete these directories.

I have patched the slaves for our most platforms with the most rapidly changing build definitions (RHEL6 and Win7) to do this, however our implementation(s) are probably not suitable for direct inclusion in Buildbot as they are platform-specific and do not include a configuration mechanism to enable/disable automatic deletion. Unfortunately we do not have time at present to put together a clean general implementation for submission.

Example trivial implementation for *NIX (not reliable on Windows due to inability to delete open files, importing shutil not shown):

for d in os.listdir(self.basedir):

• if os.path.isdir(os.path.join(self.basedir, d)):

+ path = os.path.join(self.basedir, d) + if os.path.isdir(path):

if d not in wanted_dirs:

log.msg("I have a leftover directory '%s' that is not "

• "being used by the buildmaster: you can delete "
• "it now" % d)

+ "being used by the buildmaster: deleting now" + % d) + shutil.rmtree(path)

return retval

The Windows implementation includes a shutil error handler that retries deletion up to a fixed number of attemps with a small delay between attempts, and logs cases where this approach is unsuccessful. This is necessary on Windows for a variety of reasons including security software temporarily holding files open.

[smat]

Formatting for the example implementation went awry in the description, Wiki formatted text as follows:

Example trivial implementation for *NIX (not reliable on Windows due to inability to delete open files, importing shutil not shown):
         for d in os.listdir(self.basedir):
-            if os.path.isdir(os.path.join(self.basedir, d)):
+            path = os.path.join(self.basedir, d)
+            if os.path.isdir(path):
                 if d not in wanted_dirs:
                     log.msg("I have a leftover directory '%s' that is not "
-                            "being used by the buildmaster: you can delete "
-                            "it now" % d)
+                            "being used by the buildmaster: deleting now"
+                            % d)
+                    shutil.rmtree(path)
         return retval

[dustin]

This sounds like an idea whose time has come!

I wouldn't want to turn it on by default, though, since an upgrade might then blow away important data -- for example, I keep my Buildbot source there.

Can you put together a pull request (see SubmittingPatches) that adds off-by-default configuration, allowing users both to enable this feature and to whitelist files? The configuration should go into buildbot.tac.

[krf]

I am also interested in this patch.

Or does anyone else have some code snippet to remove this on the build slaves manually?