Opened 11 years ago

Closed 10 years ago

#669 closed enhancement (fixed)

Clean up VC modes

Reported by: dustin Owned by: dustin
Priority: major Milestone: 2.0.+
Version: 0.7.12 Keywords: svn hg git cvs p4 bk darcs ship-this
Cc: dsallings, ddunbar, marcusl, zooko@…

Description (last modified by dustin)

Per an IRC conversation with ddunbar and dsal, there are really only two things users want to do with VC steps:

  • incremental updates/builds
  • full, clean builds

Different VC's can implement these options with varying degrees of reliability. What is now mode=copy works well for every supported VC *except* CVS, so mode=clobber is only really useful for CVS. Bug #632 was about a more efficient way to implement clean builds in git, that was incorrectly filed under mode=update, because it's not really a copy.

So the proposal is to have only *two* modes, common across and supported on all VCs:

  • clean
  • incremental

These will be specified with the mode= parameter. VCs can then have extra options to specify the implementation of that particular method. So Git might have mode="incremental", incremental_by="clean" *and* mode="incremental", incremental_by="copy".

Let's get this in the 0.8.0 release, since it's a pretty big behavior change (although it can be written in a backward-compatible fashion)

Related bugs:

Change History (21)

comment:1 Changed 11 years ago by maruel

Assuming the user only cares about the end result and the Source step shall do whatever is necessary to attain that goal, that means removing anything but mode=update and mode=clobber and reducing the number of parameters, especially ignore_ignores and always_purge.

I don't think there isn't much use in export. It's nice for sake of completeness but that's pretty much it. There isn't point in maintaining code that nobody uses.

We'd like to differentiate between a clobber that keeps the ignored files and directories and one that doesn't. The use case to keep the ignored files is: « For chromium's try server, we svn:ignore the output directory so we clobber the local changes but the output directory is kept. »

I don't exactly know the use case to not keep ignored files. Maybe it's not really necessary? One use case for ignoring ignored files is when switching branches.

If we keep all 3 modes; to reduce confusion, we'd change the names to:

  • incremental} is same as update. Don't try to revert any local change. Try to reuse existing checkout.
  • clean is the same as clobber ignore_ignores=False.
  • fresh is the same as clobber ignore_ignores=True.

mode=copy isn't necessary with svn anymore since correct reverting code has been added into 0.7.12. git doesn't have issue with that. I don't know about the other SCMs, for example, if mode=fresh can't be implement in CVS, it should just be implemented with the same code that mode=copy used.

comment:2 Changed 11 years ago by dustin

  • Version changed from 0.7.11 to 0.7.12

I appreciate the use-case for "clean", but I think that others might want "clean" to mean "make clean", for example.

How would you feel about using mode="incremental" and adding a VC-specific step to the build? This is how we run our builds at Zmanda -- the step after Source is ShellCommand(command=['make', 'clean'], ..).

comment:3 Changed 11 years ago by marcusl

Vaguely related: I'd like to have the option to force a build but to override the source-step (usually force a clobber on an incremental builder, just to 'reset' it).

The incremental update works most of the time, but when it fails, I currently have to clean the slave's build dir manually.

comment:4 Changed 11 years ago by dustin

  • Priority changed from critical to blocker

comment:5 Changed 11 years ago by dustin

  • Milestone changed from 0.8.0 to 0.8.+

I don't think we're going to get this fixed in 0.8.0

comment:6 Changed 11 years ago by dustin

  • Priority changed from blocker to major

comment:7 Changed 11 years ago by dustin

  • Cc marcusl added
  • Keywords svn hg git cvs p4 bk darcs monotone arch added

Related: #263, #276.

comment:8 Changed 11 years ago by dustin

  • Description modified (diff)

comment:9 Changed 11 years ago by dustin

  • Milestone changed from 0.8.+ to 0.8.1

comment:10 Changed 11 years ago by dustin

  • Description modified (diff)

comment:11 Changed 11 years ago by dustin

  • Keywords ship-this added

comment:12 Changed 11 years ago by zooko

  • Cc zooko@… added

Modern darcs stores a cache of known patches in ~/.darcs/cache and uses patches from that cache instead of downloading them from the remote host. On the bright side, that means that mode=clobber is almost exactly as fast as mode=update. On the dark side, if there is a bug in darcs or a corruption in your local filesystem (for example the patches stored in ~/.darcs/cache have been corrupted, then you can't work-around this problem by using mode=clobber!

So going forward, it might make sense for darcs users to have three options:

  1. a mode which deletes the build directory and creates a new one before doing darcs get --lazy. This is equivalent to the current mode=clobber and is nice and fast and clean. This mode should be the default.
  1. a mode like 1. but which also passes the --no-cache option to darcs. This would be useful only for working-around filesystem corruption or darcs bugs. It would be ideal if it were triggerable through the web interface so that people could try it just to check if the problem they are seeing is due to filesystem corruption or darcs bugs. (99% of the time this super-ultra-clean mode would yield the same results as the normal mode 1. so then they would know to look elsewhere for the explanation of their problems.)
  1. a mode which leaves the build directory in place, equivalent to the current mode=update. This would be a little bit faster, but not a lot, and of course it is a lot messier. I would hope that people would not use this mode, but I'm sure somebody will want it for good or bad reasons and we should provide it.

If you really want to squeeze darcs into the two-mode model, then I recommend that both modes implement 1. and there is no difference between the two modes. ;-)

comment:13 Changed 11 years ago by dustin

Zooko: to my mind you're still describing mode="clean" (1 and 2) and mode="incremental" (3), again with several implementations of the former.

For what it's worth, folks building large C++ projects have *very* good reason to want to use an incremental build.

comment:14 Changed 11 years ago by zooko

Okay, I propose that for darcs mode=clean is:

  1. a mode which deletes the build directory and creates a new one before doing darcs get --lazy. This is equivalent to the current mode=clobber and is nice and fast and clean. This mode should be the default.

and mode=incremental is:

  1. a mode which leaves the build directory in place, equivalent to the current mode=update. This would be a little bit faster, but not a lot, and of course it is a lot messier. I would hope that people would not use this mode, but I'm sure somebody will want it for good or bad reasons and we should provide it.

The darcs-specific third option would still be very useful for people who are debugging and wondering why their mode-1 clean build isn't doing what they think it should:

  1. a mode like 1. but which also passes the --no-cache option to darcs. This would be useful only for working-around filesystem corruption or darcs bugs. It would be ideal if it were triggerable through the web interface so that people could try it just to check if the problem they are seeing is due to filesystem corruption or darcs bugs. (99% of the time this super-ultra-clean mode would yield the same results as the normal mode 1. so then they would know to look elsewhere for the explanation of their problems.)

I suppose it could be implemented just by coding in a darcs-specific --no-cache flag to be added to the darcs get command.

comment:15 Changed 11 years ago by dustin

Right, and the point of this bug is that the third indented paragraph in your comment is not a "third option" - it's an alternate implementation of mode=clean. I want to change the proliferation of modes into a two-level hierarchy:

  1. mode {incremental|clean}
  2. implementation (arbitrary per VC)

and ideally you could specify the implementation for each mode independently, so that when the admin forces a mode change e.g., through the web interface, it would still use the desired implementation.

Git and SVN both have a proliferation of implementations of each mode, and to date these have been swapped in willy-nilly (on the basis that the new way is obviously superior, even though it's often not appropriate or optimal for existing users).

Thanks for the feedback wrt darcs!

comment:16 Changed 11 years ago by dustin

  • Description modified (diff)

comment:17 Changed 11 years ago by dustin

  • Description modified (diff)

comment:18 Changed 11 years ago by dustin

  • Keywords monotone arch removed

comment:19 Changed 11 years ago by dustin

  • Milestone changed from 0.8.2 to 0.8.3

bumping to 0.8.3

comment:20 Changed 10 years ago by dustin

  • Milestone changed from 0.8.3 to 1.0.+

comment:21 Changed 10 years ago by dustin

  • Resolution set to fixed
  • Status changed from new to closed

This was largely implemented this summer by in3xes, yay!

Note: See TracTickets for help on using tickets.