Opened 8 years ago

Closed 7 years ago

#1062 closed support-request (fixed)

buildstep subclass of SVN step broken by 0.82 (worked with 0.81)

Reported by: aclight Owned by:
Priority: major Milestone: ongoing
Version: 0.8.2 Keywords: svn
Cc: aclight@…

Description

I have a custom buildstep class that is a subclass of the SVN buildstep provided by buildbot. Before today I was running buildbot 0.81 on my Ubuntu Linux master server and buildbot-slave 0.81 on a Windows and Macintosh slave. After upgrading to version 0.82 on both the master and slave, and after executing buildbot upgrade-master . and the equivalent command on the slaves, my custom buildstep fails with an exception.

The twistd version on master and all slaves is 10.1.0. The master's Python version is 2.6.4, and on the slave I'm focusing on is 2.5.1.

In my BuildFactory? subclass, I have the following code:

...
steps = []
steps.append(SVN(baseURL= svn_base_path + 'src/', defaultBranch=src_dir, workdir=src_workdir, username=config_svn_username, password=config_svn_password, alwaysUseLatest=True, mode='update'))
steps.append(SVN_HEAD_revision(svnurl= svn_base_path + 'sandbox/adam/projects/' + str(self.os) + '/myapp' + str(self.myapp_version) + '/', workdir=project_workdir))
...

The various variables used in those lines (eg. src_dir, src_workdir, etc.) are properly defined earlier within the class.

Here is the entire code of my SVN_HEAD_revision class:

class SVN_HEAD_revision(SVN):
  """A class based on SVN that always checks out from HEAD, regardless of
  what revision was checked out of another repository."""

  def __init__(self, alwaysUseLatest=True, retry=(60,5),
    username=config_svn_username, password=config_svn_password,
    **kwargs):
    """
    See documentation of parent SVN and Source classes for explanations
    of these parameters.
    """
    # Store arguments that aren't directly used by this
    # class into kwargs so the base class(es) will
    # see and use them.
    kwargs['alwaysUseLatest'] = alwaysUseLatest
    kwargs['retry'] = retry
    kwargs['username'] = username
    kwargs['password'] = password
    
    # This class must always be initialized with svnurl instead
    # of the alternative of using baseURL and the branch
    # that triggered this build.  This is because
    # the code that's checked out using this class
    # is not the application's source code and therefore the branch
    # or revision of the actual change to the application's source
    # code does not apply.
    self.branch = None
    self.defaultBranch = None
    kwargs['defaultBranch'] = None

    SVN.__init__(self, **kwargs)
    self.branch = None
    self.defaultBranch = None
    self.addFactoryArguments(
      alwaysUseLatest=alwaysUseLatest,
      retry=retry,
      username=username,
      password=password,
      defaultBranch=self.defaultBranch,
     )

  def computeSourceRevision(self, changes):
    return 'HEAD'

  # NOTE:  We override start() so that we can force
  # computeSourceRevision to always be called
  # regardless of the value of revision.
  # This is necessary because the revision
  # numbers of the application's source repository and
  # the other repositories code is checked out from
  # (such as sandbox, testing, etc.) will not be the same.
  def start(self):
    print "SVN_HEAD_revision start called"
    if self.notReally:
      log.msg("faking %s checkout/update" % self.name)
      self.step_status.setText(["fake", self.name, "successful"])
      self.addCompleteLog("log",
                          "Faked %s checkout/update 'successful'
" 
                          % self.name)
      return SKIPPED

    # what source stamp would this build like to use?
    s = self.build.getSourceStamp()
    # Because this class should only be used for SVN source
    # code in a different repository or location than the main
    # source code of the application, we always want to set branch to None
    # so that the application's source code branch that's associated
    # with this step isn't used.
    branch = None

    # Always compute what revision should be used.
    revision = self.computeSourceRevision(s.changes)
    
    print "revision being used is:" + revision
    print "workdir is:" + self.workdir
    
    # if patch is None, then do not patch the tree after checkout

    # 'patch' is None or a tuple of (patchlevel, diff)
    patch = s.patch
    if patch:
      self.addCompleteLog("patch", patch[1])

    self.startVC(branch, revision, patch)

The first step, using the SVN class, is successful. The second, using my custom SVN_HEAD_revision class, fails.

I have included the twistd.log records for the build on both the master and slave. For the time period, I haven't deleted any lines. I have replaced some text with XXXX and the like.

Here is the relevant portion of the twistd.log file on the master:

	2010-11-17 15:12:45-0800 [Broker,1,192.168.1.27] <Build i6_mac105>.startBuild
2010-11-17 15:12:45-0800 [Broker,1,192.168.1.27] ShellCommand.startCommand(cmd=<RemoteCommand 'svn' at 60728240>)
2010-11-17 15:12:45-0800 [Broker,1,192.168.1.27]   cmd.args = {'username': 'XXXXXX', 'password': 'XXXXXX', 'workdir': 'sources/myapp_source', 'patch': None, 'retry': None, 'mode': 'update', 'timeout': 1200, 'svnurl': 'svn://XXX.XXX.XXX.XXX/src/myapp5px/trunk', 'always_purge': None, 'revision': None}
2010-11-17 15:12:45-0800 [Broker,1,192.168.1.27] <RemoteCommand 'svn' at 60728240>: RemoteCommand.run [0]
2010-11-17 15:12:45-0800 [Broker,1,192.168.1.27] LoggedRemoteCommand.start
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] <RemoteCommand 'svn' at 60728240> rc=0
2010-11-17 15:12:47-0800 [-] closing log <buildbot.status.builder.LogFile instance at 0x3bd4200>
2010-11-17 15:12:47-0800 [-] releaseLocks(<buildbot.steps.source.SVN instance at 0x3aeffc8>): []
2010-11-17 15:12:47-0800 [-]  step 'svn' complete: success
2010-11-17 15:12:47-0800 [-] SVN_HEAD_revision start called
2010-11-17 15:12:47-0800 [-] revision being used is:HEAD
2010-11-17 15:12:47-0800 [-] workdir is:sources/myapp_source/Xcode
2010-11-17 15:12:47-0800 [-] ShellCommand.startCommand(cmd=<RemoteCommand 'svn' at 65336168>)
2010-11-17 15:12:47-0800 [-]   cmd.args = {'username': 'XXXXXX', 'password': 'XXXXXX', 'retry': (60, 5), 'patch': None, 'mode': 'update', 'timeout': 1200, 'svnurl': 'svn://XXX.XXX.XXX.XXX/sandbox/adam/projects/mac/myapp6/', 'always_purge': None, 'revision': 'HEAD'}
2010-11-17 15:12:47-0800 [-] <RemoteCommand 'svn' at 65336168>: RemoteCommand.run [1]
2010-11-17 15:12:47-0800 [-] LoggedRemoteCommand.start
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] BuildStep.failed, traceback follows
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] Unhandled Error
	Traceback (most recent call last):
	Failure: exceptions.KeyError
	
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] addHTMLLog(err.html)
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] addCompleteLog(err.text)
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] releaseLocks(<myapp_tests.SVN_HEAD_revision instance at 0x39ee050>): []
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27] BuildStep.failed now firing callback
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27]  step 'svn_1' complete: exception
2010-11-17 15:12:47-0800 [Broker,1,192.168.1.27]  <Build i6_mac105>: build finished

Here is the relevant portion of the twistd.log file on the slave:

2010-11-17 15:12:45-0800 [Broker,client] SlaveBuilder.remote_print(i6_mac105): message from master: ping
2010-11-17 15:12:45-0800 [Broker,client]  startCommand:svn [id 0]
2010-11-17 15:12:45-0800 [Broker,client] RunProcess._startCommand
2010-11-17 15:12:45-0800 [Broker,client]  /usr/bin/svn update --non-interactive --no-auth-cache --username XXXX --password XXXX --revision HEAD
2010-11-17 15:12:45-0800 [Broker,client]   in dir /BuildSlave/myapp/i6_mac105/sources/myapp_source (timeout 1200 secs)
2010-11-17 15:12:45-0800 [Broker,client]   watching logfiles {}
2010-11-17 15:12:45-0800 [Broker,client]   argv: ['/usr/bin/svn', 'update', '--non-interactive', '--no-auth-cache', '--username', 'XXXX', '--password', 'XXXX', '--revision', 'HEAD']
2010-11-17 15:12:45-0800 [Broker,client]  environment: {'TERM_PROGRAM_VERSION': '240.2', 'LOGNAME': 'XXXX', 'USER': 'XXXX', 'HOME': '/Users/buildbot', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin', 'DISPLAY': '/tmp/launch-LvRqYa/:0', 'TERM_PROGRAM': 'Apple_Terminal', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm-color', 'Apple_PubSub_Socket_Render': '/tmp/launch-GyM4LE/Render', 'SHLVL': '1', 'SECURITYSESSIONID': 'db6e50', '_': '/usr/local/bin/buildslave', 'MANPATH': '/usr/share/man:/usr/local/share/man:/usr/X11/man:/usr/local/git/share/man', 'SSH_AUTH_SOCK': '/tmp/launch-kJWYe3/Listeners', 'SHELL': '/bin/bash', 'TMPDIR': '/var/folders/kK/kKGKhUxkGVWpICsbhbdU3k+++TU/-Tmp-/', 'LC_MESSAGES': 'C', 'OLDPWD': '/BuildSlave/myapp/i6_mac105', '__CF_USER_TEXT_ENCODING': '0x1F8:0:0', 'PWD': '/BuildSlave/igor/i6_mac105/sources/myapp_source', 'COMMAND_MODE': 'unix2003'}
2010-11-17 15:12:45-0800 [Broker,client]   closing stdin
2010-11-17 15:12:45-0800 [Broker,client]   using PTY: False
2010-11-17 15:12:47-0800 [-] command finished with signal None, exit code 0, elapsedTime: 1.855041
2010-11-17 15:12:47-0800 [-] RunProcess._startCommand
2010-11-17 15:12:47-0800 [-]  /usr/bin/svnversion .
2010-11-17 15:12:47-0800 [-]   in dir /BuildSlave/myapp/i6_mac105/sources/myapp_source (timeout 1200 secs)
2010-11-17 15:12:47-0800 [-]   watching logfiles {}
2010-11-17 15:12:47-0800 [-]   argv: ['/usr/bin/svnversion', '.']
2010-11-17 15:12:47-0800 [-]  environment: {'TERM_PROGRAM_VERSION': '240.2', 'LOGNAME': 'XXXX', 'USER': 'XXXX', 'HOME': '/Users/buildbot', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin', 'DISPLAY': '/tmp/launch-LvRqYa/:0', 'TERM_PROGRAM': 'Apple_Terminal', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm-color', 'Apple_PubSub_Socket_Render': '/tmp/launch-GyM4LE/Render', 'SHLVL': '1', 'SECURITYSESSIONID': 'db6e50', '_': '/usr/local/bin/buildslave', 'MANPATH': '/usr/share/man:/usr/local/share/man:/usr/X11/man:/usr/local/git/share/man', 'SSH_AUTH_SOCK': '/tmp/launch-kJWYe3/Listeners', 'SHELL': '/bin/bash', 'TMPDIR': '/var/folders/kK/kKGKhUxkGVWpICsbhbdU3k+++TU/-Tmp-/', 'LC_MESSAGES': 'C', 'OLDPWD': '/BuildSlave/myapp/i6_mac105', '__CF_USER_TEXT_ENCODING': '0x1F8:0:0', 'PWD': '/BuildSlave/myapp/i6_mac105/sources/myapp_source', 'COMMAND_MODE': 'unix2003'}
2010-11-17 15:12:47-0800 [-]   closing stdin
2010-11-17 15:12:47-0800 [-]   using PTY: False
2010-11-17 15:12:47-0800 [-] command finished with signal None, exit code 0, elapsedTime: 0.111992
2010-11-17 15:12:47-0800 [-] SlaveBuilder.commandComplete <buildslave.commands.svn.SVN instance at 0x5d83c8>
2010-11-17 15:12:47-0800 [Broker,client] Peer will receive following PB traceback:
2010-11-17 15:12:47-0800 [Broker,client] Unhandled Error
	Traceback (most recent call last):
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/banana.py", line 153, in gotItem
	    self.callExpressionReceived(item)
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/banana.py", line 116, in callExpressionReceived
	    self.expressionReceived(obj)
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/pb.py", line 514, in expressionReceived
	    method(*sexp[1:])
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/pb.py", line 826, in proto_message
	    self._recvMessage(self.localObjectForID, requestID, objectID, message, answerRequired, netArgs, netKw)
	--- <exception caught here> ---
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/pb.py", line 840, in _recvMessage
	    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
	  File "/Library/Python/2.5/site-packages/Twisted-10.1.0-py2.5-macosx-10.5-i386.egg/twisted/spread/flavors.py", line 114, in remoteMessageReceived
	    state = method(*args, **kw)
	  File "/Library/Python/2.5/site-packages/buildbot_slave-0.8.2-py2.5.egg/buildslave/bot.py", line 123, in remote_startCommand
	    self.command = factory(self, stepId, args)
	  File "/Library/Python/2.5/site-packages/buildbot_slave-0.8.2-py2.5.egg/buildslave/commands/base.py", line 117, in __init__
	    self.setup(args)
	  File "/Library/Python/2.5/site-packages/buildbot_slave-0.8.2-py2.5.egg/buildslave/commands/svn.py", line 28, in setup
	    SourceBaseCommand.setup(self, args)
	  File "/Library/Python/2.5/site-packages/buildbot_slave-0.8.2-py2.5.egg/buildslave/commands/base.py", line 227, in setup
	    self.workdir = args['workdir']
	exceptions.KeyError: 'workdir'

I have noticed that in the log output from the master, the first svn command, which executes successfully, lists workdir as one of the members of cmd.args, while the second command, which fails, does not. But where I create the build step, I am passing workdir as a parameter.

I would appreciate any help that anyone can provide.

Change History (4)

comment:1 Changed 8 years ago by aclight

  • Cc aclight@… added

In case anyone wonders why I have my custom SVN_HEAD_revision class, it's because for each build, I need to have the most recent revision of the application's source code checked out, and I also need to have the HEAD revision of a few other things in different repositories checked out (including tests, project files, etc.). As far as I could tell, it wasn't possible to do this without writing my own custom class.

comment:2 Changed 8 years ago by dustin

  • Keywords svn added
  • Type changed from defect to support-request

I would start by comparing your start impl with that of the SVN steps in 0.8.2.

You've run across a fundamental problem with Buildbot that will take some time to fix. Jacob summed it up nicely at http://jacobian.org/writing/buildbot/ci-is-hard/: Buildbot is a framework, not an app. But Buildbot doesn't have a well-defined API. We've never written down what you should and should not override in a buildstep, much less a Source buildstep. If we tried to write such a thing down, it would be highly contentious and probably still break a lot of custom configurations.

comment:3 Changed 7 years ago by ayust

  • Milestone changed from undecided to ongoing

comment:4 Changed 7 years ago by aclight

  • Resolution set to fixed
  • Status changed from new to closed

For what it's worth, I was able to fix this problem. I did so by changing my SVN_HEAD_revision::init method to the following.

class SVN_HEAD_revision(SVN):
  """A class based on SVN that always checks out from HEAD, regardless of
  what revision was checked out of another repository."""

  def __init__(self, alwaysUseLatest=True, retry=(60,5),
    username=config_svn_username, password=config_svn_password,
    **kwargs):
    """
    See documentation of parent SVN and Source classes for explanations
    of these parameters.
    """
    # Store arguments that aren't directly used by this
    # class into kwargs so the base class(es) will
    # see and use them.
    kwargs['alwaysUseLatest'] = alwaysUseLatest
    kwargs['retry'] = retry
    kwargs['username'] = username
    kwargs['password'] = password
    
    # I found that these lines
    # were necessary when I do self.args['workdir'] = properties.render(self.workdir)
    # in this class's start() implementation, though I don't understand why.
    # This will get added to args later, after properties are rendered
    if kwargs['workdir']:
      self.workdir = kwargs['workdir']
    
    # This class must always be initialized with svnurl instead
    # of the alternative of using baseURL and the branch
    # that triggered this build.  This is because
    # the code that's checked out using this class
    # is not the application's source code and therefore the branch
    # or revision of the actual change to the application source
    # code does not apply.
    self.branch = None
    self.defaultBranch = None
    kwargs['defaultBranch'] = None

    SVN.__init__(self, **kwargs)
    self.branch = None
    self.defaultBranch = None
    self.addFactoryArguments(
      alwaysUseLatest=alwaysUseLatest,
      retry=retry,
      username=username,
      password=password,
      defaultBranch=self.defaultBranch,
     )

The change was the addition of the following lines which I copied from Source::init:

    if kwargs['workdir']:
      self.workdir = kwargs['workdir']

I don't understand why those lines were necessary but probably it is due to my lack of Python knowledge.

In any case, thanks for your help.

Note: See TracTickets for help on using tickets.