Opened 7 years ago

Last modified 4 years ago

#2061 new enhancement

patch: make P4Poller pick up where it left off

Reported by: dberger Owned by:
Priority: patches-accepted Milestone: 0.9.+
Version: 0.8.4p2 Keywords: p4, sprint
Cc: bdbaddog

Description

$ p4 diff -du ../external/buildbot-0.8.4p1/...
--- //main/tools/external/buildbot-0.8.4p1/buildbot/changes/p4poller.py	2011-06-28 16:38:20.000000000 -0700
+++ /home/buildbot/buildbot/external/buildbot-0.8.4p1/buildbot/changes/p4poller.py	2011-06-28 16:38:20.000000000 -0700
@@ -44,8 +44,10 @@
     """This source will poll a perforce repository for changes and submit
     them to the change master."""
 
     compare_attrs = ["p4port", "p4user", "p4passwd", "p4base",
-                     "p4bin", "pollInterval"]
+                     "p4bin", "pollInterval", 
+                     "split_file"]
 
     env_vars = ["P4CLIENT", "P4PORT", "P4PASSWD", "P4USER",
                 "P4CHARSET"]
@@ -92,6 +94,35 @@
 
     @defer.deferredGenerator
     def _poll(self):
+        if self.last_change is None:
+            # try asking the master for it's most recent processed 
+            # change, turn that into a p4 change number, and start 
+            # there
+            try:
+                log.msg('P4Poller: asking buildmaster for latest changeid')
+                wfd = defer.waitForDeferred( 
+                    self.master.db.changes.getLatestChangeid() )
+                yield wfd
+                latest_change_id =  wfd.getResult()
+                log.msg('P4Poller: asking buildmaster about changeid = %s' % str(latest_change_id) )
+                wfd = defer.waitForDeferred(
+                    self.master.db.changes.getChange( latest_change_id ) )
+                yield wfd
+                chDict = wfd.getResult()
+                revision = int( chDict.get( 'revision', -1 ) )
+                when_timestamp = int( chDict.get( 'when_timestamp', 0 ) )
+                now = time.time()
+                log.msg('P4Poller: last saw revision %d on %s' % ( revision,
+                                                                   time.ctime( when_timestamp ) ) )
+                if now - when_timestamp < 0 or now - when_timestamp > 60*24:
+                    log.msg('P4Poller: last seen revision was in the future, or more than a day ago, ignoring')
+                else:
+                    self.last_change = int( chDict.get( 'revision', None ) )
+
+            except Exception, e:
+                print "couldn't determine last change, starting with the next one: ", e
+
         args = []
         if self.p4port:
             args.extend(['-p', self.p4port])
@@ -124,6 +155,16 @@
                 log.msg('P4Poller: starting at change %d' % num)
                 self.last_change = num
                 return
+
+            if ( num - last_change ) < 0 or ( num - last_change ) > 10000:
+                # if we suddenly go negative, or we've seen a crazy number 
+                # of changes, start on the change we just saw
+                log.msg('P4Poller: saw a ridiculous number of changes, starting at change %d' % num)
+                self.last_change = num
+                return
+
             changelists.append(num)
         changelists.reverse() # oldest first

Attachments (2)

p4poller.patch (3.1 KB) - added by dberger 7 years ago.
bug2061.patch (3.8 KB) - added by dustin 7 years ago.
bug2061.patch

Download all attachments as: .zip

Change History (11)

comment:1 Changed 7 years ago by dberger

By way of explanation - I finally got tired of having to restart buildbot to change our P4Poller split_file, so I added split_file to the compared attrs for the class, which then caused the source to be re-built on reconfig (good) but for the poller to lose it's place (bad). so here's a fix that's "working for us."

Last edited 7 years ago by dberger (previous) (diff)

comment:2 Changed 7 years ago by dberger

+ when_timestamp = int( chDict.get( 'when_timestamp', 0 ) )

should be

+ when_timestamp = chDict.get( 'when_timestamp', 0 )

and

+ time.ctime( when_timestamp ) ) ) should be

+ when_timestamp.ctime() ) )

I apparently had the unfixed version in my clipboard when I pasted.

comment:3 Changed 7 years ago by dberger

Arg, there was at least one more change - how 'bout I just regenerate the working diff and attach it.

Changed 7 years ago by dberger

comment:4 Changed 7 years ago by dustin

Heh, I assume that's P4's spin on a "patch"? :)

I'll get it merged..

Changed 7 years ago by dustin

bug2061.patch

comment:5 Changed 7 years ago by dustin

Here's an updated version of the patch. However, this doesn't pass the tests anymore, so I didn't merge it to buildbot.

Also, please use log.msg instead of print.

comment:6 Changed 7 years ago by dustin

  • Keywords p4 added
  • Milestone changed from undecided to 0.8.+

comment:7 Changed 6 years ago by tom.prince

  • Cc bdbaddog added
  • Priority changed from major to patches-accepted

The proper way to handle this, is to store some state in the database. The current gitpoller has a reasonable example of how to do this.

comment:8 Changed 6 years ago by dustin

  • Keywords sprint added

comment:9 Changed 4 years ago by dustin

  • Milestone changed from 0.8.+ to 0.9.+

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.