Opened 10 years ago

Closed 5 years ago

#438 closed defect (fixed)

Mercurial does not update to the right revision when building a tag from a named branch

Reported by: cdevienne Owned by: Callek
Priority: major Milestone: 0.8.+
Version: 0.8.5 Keywords: hg
Cc: Pike, marcusl

Description (last modified by dustin)

This morning I ran into a very big issue with the way buildbot do a Mercurial checkout.

Here is the context :

  • The repository contains a named branch
  • A tag has been set on a revision of that branch
  • From the buildbot web ui, we force a build with the tag as the revision
  • buildbot do a "hg clone --rev THETAG <url>"
  • The actual current changeset of the clone after checking out is NOT the revision corresponding to the tag. Instead it update to the latest revision that is on the "default" branch.

In the end the build is not done on the correct revision.

I suggest not to rely on the "clone --rev" but rather do a "pull" followed by a "update -r". Additional reasons for that :

  • It will be way faster for big repositories (ours is ~130M).
  • When using clone --rev, the tag is "lost" because the .hgtags file is set in a later changeset than the tagged changeset.

Note that I have reproduced the issue with buildbot 0.7.9 and 0.7.10, and Mercurial 1.0.1 and 1.1.2

Change History (15)

comment:1 Changed 10 years ago by Pike

  • Cc Pike marcusl added

I don't see references to 'clone' or 'pull' --rev in the 0.7.10 code. AFAICT, it's doing a init, pull, and then a update --rev

Can you show what it's precisely doing on .10?

comment:2 Changed 10 years ago by cdevienne

I realize that only the serveur side was 0.7.10.

I will check with a .10 on the client side too asap.

comment:3 Changed 10 years ago by cdevienne

  • Resolution set to worksforme
  • Status changed from new to closed

Problem gone with .10 on the client side.

Sorry for the noise, I am closing the ticket right now.

Christophe

comment:4 Changed 7 years ago by marcusl

  • Resolution worksforme deleted
  • Status changed from closed to reopened
  • Version changed from 0.7.10 to 0.8.5

This happens again in 0.8.5 (both slave and master):

  • I press "Force build" with revision set to "v0.2.1"
  • Slave does 'hg.EXE' 'clone' '--verbose' '--noupdate' '--rev' u'v0.2.1' '<repo url>' 'build' which gets the rev, but misses the changeset that created the tag
  • Then it tries to update to tag 'hg.EXE' 'update' '--clean' '--repository' 'build' '--rev' u'v0.2.1' and fails with:
> **abort: unknown revision 'v0.2.1'! **
Last edited 7 years ago by dustin (previous) (diff)

comment:5 Changed 7 years ago by dustin

Marcus, what command *should* it run? It seems like the --rev should do the trick, right?

comment:6 Changed 7 years ago by marcusl

Dustin: It would seem so, for the causal observer. :)

However, due to the (slightly bone-headed, IMO) way tags are handled in Mercurial, the info about the tag itself is not preserved when cloning with '--rev', so the tag-info will not exist in the clone, even though the top revision is the one that was tagged.

Tags are handled by a .hgtags, which is version controlled too, and since the tags are created in that file in a revision made after the revision being tagged (

Basically:

  • Revision A is created
  • Revision B adds tag X to revision A
  • If "clone --rev = X" is executed, the clone contains rev A, but not B, so it does not know about tag X, so "update X" fails, as do my scripts for detecting the current version from vc tags)

There are a few ways to tackle this:

  • Always clone everything (easy, but potentially wasteful if a very old tag is being checked out)
  • Detect the revision B where tag was made, clone that one (tricky to implement, but certainly possible. I've seen code for this on stackoverflow.com )
  • Add tag locally using "hg tag X --local" (potentially dangerous due to conflicts with later updates, unless tag is removed after build or clone is not re-used"

This has been discussed in several places, and the root issue has not been fixed yet. There is some resistance to fixing this in the Mercurial dev team, obviously for a few good reasons, and maybe because no fully superior solution is available, yet.

comment:7 Changed 7 years ago by dustin

Interesting. I think the only option for Buildbot will be the first - "always clone everything". How can we do that? Just omit the --rev? How can we know that the requested revision is a tag and not a branch or a revision string?

Coincidentally, I'm helping to debug failures in Mozilla where clones are taking longer than an hour, precisely because we have to clone everything and can't do shallow clones. Mercurial's got some growing pains!

comment:8 Changed 7 years ago by marcusl

Yes, just omitting the rev will work, which is what the OP said too.

  • A revision string will either be an int, a 12-char hex, or a 40-char hex.
  • Branch names could be anything though, so that's trickier.

One could, of course, run 'hg tags' or 'hg branches' against the remote repo and see if it's included in the result.

I agree with the growing pains thing. I think we should really look at repo-caching on the slave(s), in order to reduce this (something I wanted to do with SVN too, btw), or just wait for http://mercurial.selenic.com/wiki/ShallowClone to be done. :)

comment:9 follow-up: Changed 7 years ago by dustin

So it sounds like users running into this will need to use mode=full all the time, since we can't reliably tell when --rev will work and when it won't.

The fallback behavior should take care of that, though, right?

comment:10 Changed 7 years ago by dustin

  • Keywords hg added
  • Milestone changed from undecided to 0.8.+

comment:11 Changed 6 years ago by tom.prince

  • Owner set to Callek
  • Status changed from reopened to assigned

comment:12 in reply to: ↑ 9 Changed 5 years ago by pepsiman

Replying to dustin:

So it sounds like users running into this will need to use mode=full all the time, since we can't reliably tell when --rev will work and when it won't.

If using --rev is an optimisation, then I'd much prefer something that works slowly than something that doesn't work at all.

comment:13 Changed 5 years ago by dustin

  • Description modified (diff)

I don't think it's an optimization, here -- what's the slow, effective way? Apologies if I've forgotten much of the context here!

comment:14 Changed 5 years ago by pepsiman

The mercurial steps can operate in full or incremental mode.

full mode can use clean, fresh or clobber methods.

The clean and fresh methods currently use "pull --rev rev".

Using "pull --rev rev" might sometimes be faster than "pull", but it does the wrong thing when given a tag.

I'd like to change from this:

full incremental
clobber clean fresh None
rmdir purge purge --all  
clone pull --rev rev pull
update --clean --rev rev

to this:

full incremental
clobber clean fresh None
rmdir purge purge --all  
clone pull
update --clean --rev rev

Patch:

--- a/master/buildbot/steps/source/mercurial.py
+++ b/master/buildbot/steps/source/mercurial.py
@@ -226,8 +226,6 @@ class Mercurial(Source):
 
     def _pullUpdate(self, res):
         command = ['pull' , self.repourl]
-        if self.revision:
-            command.extend(['--rev', self.revision])
         d = self._dovccmd(command)
         d.addCallback(self._checkBranchChange)
         return d

comment:15 Changed 5 years ago by pepsiman

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.