Add a Source step that get a tar.gz archive from an FTP server and uncompress it, ready to be build.

[dtrosset]

I've always wanted to get this kind of source step. Instead of connecting to a VC server and perform a checkout/update, it connects to an FTP server, downloads a source archive from it, and uncompresses it in the indicated build (workdir) directory.

If the source archive placed all its files in a subdirectory (usually source-X.y.z), this portion of the extracted file names can be removed (as an option). As such, the usual and very useful file source-X.y.z/Makefile is extracted directly to build/Makefile directory. allowing easy further Compile steps, as with other source steps.

The patch attached contains the master and slave changes, plus the documentation addition.

[dtrosset]

FtpTar? source step patch

[warner]

I think there's some good discussion taking place on this one in the mailing list, but I'll ask the question here too: why make this a new step? Why not invoke ftp or wget or curl or some other convenient command-line tool, then invoke 'tar xf' on the file?

I can see how this would make life easier for a windows buildslave that doesn't have those tools already installed, but I'm not sure this is worth having this extra single-purpose code inside buildbot. FTP (versus http, or scp) and gzip (versus bz2 or zip) and tar (versus zip or some other aggregation format) is not very general. It might be reasonable to make it more general (i.e. have a trio of 'get file from server onto the slave' step, ftp/http/scp, and a 'unpack file' step that knows about .zip or .tar.gz or whatever), but of course then we're adding 4 or 5 buildsteps instead of just one.

It is possible to run ShellCommands outside the normal build/ subdirectory, by providing an alternate workdir= argument. For example:

 f.addStep(ShellCommand(workdir=".", command=["wget", "ftp://blah/foo.tar.gz"]))
 f.addStep(ShellCommand(workdir=".", command=["tar", "xf", "foo.tar.gz"]))
 f.addStep(ShellCommand(workdir=".", command=["mv", "foo", "build"]))
 f.addStep(Compile)

Tell me more about the use case for this feature.. let's see if adding a new buildstep is the right way to go, or maybe improving the docs or adding some convenience features to other steps might be better.

[dtrosset]

As I explained on the mailing list, all of the Source steps provide management of the build directory. You solved it in a single command (mv) for UNIX, but you would require at least 2 more on Windows. And frankly speaking, using a ShellCommand to issue a mv is overkill. Just think about the waterfall page having a (hopefully green) box for mv. And think also to all that lies behind for it, such as the log files.

I am currently preparing another version. I finally split the FtpTar command into two. There's the FtpDownload step, which is a Source step, and takes care of the build directory. It is now implemented using twisted. As such it is self sufficient for FTP and can be used standalone. But it is meant to be used with ...

The Uncompress step, that is a ShellCommand with some name, description definitions, and a special parameter filename that generates automatically the command.

I'd have liked to keep these two in a single step. I mean having the FTP download, plus the uncompress step integrated, the latter kept in the form of a ShellCommand step (managing all of the uncompression possibilities would be really too much work).

[dtrosset]

Separate FtpDownload? and Uncompress steps (from pristine 0.7.8, i.e. not from FtpTar?)

[dustin]

I'm still leaning toward warner's earlier concerns with this step. We both see a future for buildbot where the slave only knows how to do a few things, but does them well:

• run a shell command (with logging, env vars, etc.)
• download a file
• upload a file

That this inclination creates extra noise in the waterfall display is really a bug in the display mechanism, and in particular in the equal weighting of all steps in the display. BuildbotOneOh? will hopefully solve this problem.

I'll leave it to warner to decide whether this should go into the next release, in the interim.

[dustin]

I agree with myself, only more strongly. I do not want to add slave code to support this.