New build step for uploading directories from slave to master: DirectoryUpload

[mue]

As mentioned on the mailing list (27. November 2008), I've made an DirectoryUpload? step. It creates all needed directories on the master and transfers empty directories as well. Maybe it looks a little bit ugly, because it is the first time, that I work with twisted. DirectoryUpload? is based on FileUpload?, because of my missing experience with twisted.

Any comments are welcome! Hopefully it gets in the right shape before the next release.

[mue]

missing second part

[mue]

I just noticed a problem, sometimes my step miss some files. I will upload a fixed patch, when available.

[dustin]

This also needs a test or two and some documentation, while you're at it. Then keyword it "review" and I'll take a look.

[dustin]

Any update on this step?

In general, I'm disinclined to add new functionality on the slave side. Unless there's a convincing use-case in which uploading e.g., a tarball or a zip file is not appropriate, I'd rather not incorporate this functionality. As always, I'm happy to be overridden by warner.

One thing that may help my solution (uploading an archive file) is the ability to run a shell command on the master -- this would let you untar/unzip the uploaded archive wherever you needed it on the master.

[mue]

Replying to dustin:

Any update on this step?

I had no free time over christmas, but hopefully I have some time over the next weeks.

In general, I'm disinclined to add new functionality on the slave side. Unless there's a convincing use-case in which uploading e.g., a tarball or a zip file is not appropriate, I'd rather not incorporate this functionality. As always, I'm happy to be overridden by warner.

My usecase is to upload the documentation to the master and publish it with the help of buildbots webserver. The documentation isn't a single file as described in the buildbot-manual, but an directory with many files and subdirectories (generated by doxygen/epydoc). I would need nearly hundred steps to upload every single file.

One thing that may help my solution (uploading an archive file) is the ability to run a shell command on the master -- this would let you untar/unzip the uploaded archive wherever you needed it on the master.

Well, the ability to run a shell command on the master would definitly help on this. Then I can upload an archive file.

If DirectoryUpload? is not the preferred way, but a MasterShellCommand? is, then I will stop working on this. But I'm unsure if I can help with the MasterShellCommand? because of my missing experience with buildbot/twisted. I have already a hard time with DirectoryUpload?.

[mue]

Ok, I remade my patch completely (Part 2 is not needed anymore, but I can't delete it).

I did a new start with some tests and finally got it running. It has not the complexity of FileUpload?, no upload in parts or sizelimit. This is just because I'm not familiar enough with twisted. I had a hard time to get it working like this.

What documentation is needed to get a review?

[dustin]

I like this, but I actually have a more general solution that I'm using locally, which I would prefer to add to the distro. I won't have time to get that into 0.7.10, though, so I'm going to move this to the 0.7.+ milestone.

[mue]

I made a patch for the documentation. It is relativ short, but the main part is described under FileUpload?. I tag this review now.

[dustin]

Created commit 4573cc2: (refs #393) add DirectoryUpload
 4 files changed, 578 insertions(+), 24 deletions(-)