wiki:MetaBuildbotConfig

Master

The buildbot master is configured in ~/metabuildbot. It runs a slaveport on port 9989 and web on port 8010. It is set up to run from the source checkout in ~/metabuildbot/src.

To start/stop/restart the master, use e.g.,

buildbot reconfig ~/metabuildbot/

Config

The Buildbot config itself is contained in  https://github.com/buildbot/metabbotcfg, checked out at ~/metabuildbot/metabbotcfg, with a master.cfg symlink pointing into the directory. Note that passwords are contained in the metabbotcfg directory, but are not stored in the git repository.

The builder configuration flows from the slaves. There are three types of builders: compatibility, slave-specific, and purpose-specific. The compatibility builders run a particular version of Python and Twisted to check compatibility. Slaves are tagged with the versions of Python they support, and the builders use virtualenv to install various versions of Twisted. For most of these builders, several slaves are available, allowing better parallelism for faster results.

The slave-specific builders attempt to build on a specific slave. This is usually to identify issues unique to that slave, e.g., certain versions of Windows or older SQLite versions in enterprise linux distributions.

Finally, the purpose-specific builders do things like generate documentation, run lint-like tools, or create a coverage analysis.

Virtualenvs

Most of the builds use a virtualenv to contain the Buildbot install; these use a custom step (defined in builders.py) as well as a cache of Python packages at http://buildbot.net/pkgs.

Slaves

Cygwin slave setup

  1. Installed cygwin 1.7 with:
    • curl
    • cvs
    • gcc4
    • gcc4-g++
    • git
    • libsqlite3-devel
    • mercurial
    • nano
    • python
    • sqlite3
    • subversion
    • (optional)
      • make
      • openssh
      • screen
      • vim
  2. Setup Tools (easy_install)
  3. Install python dependencies
    • easy_install zope.interface
    • easy_install twisted
    • easy_install Jinja2
    • easy_install buildbot
    • easy_install simplejson
    • easy_install pysqlite
    • easy_install mock
  4. Create and start slave
    • mkdir ~/slave
    • cd ~/slave
    • buildbot create-slave . host:port slavename pwd
    • make start
  5. If you want to have the user auto-login
    1. run "control userpasswords2"
    2. Uncheck "User must enter username and password to use this computer", click OK and enter the user credentials
    3. Add a shortcut in startup to 'c:cygwininash.exe --login -c "cd slave && make start log"'
    4. Reboot

Pure Windows setup

These are probably insufficient, for instance I seems I forgot to include the easy_install step. But easy_install is really fucked up on Windows so it's probably better to not even try this route.

  1. Python 2.6
  2. Update PATH
    • WinKey-Pause
    • Advanced Settings
    • Environment variables
    • Add to PATH: ;c:python26;c:python26scripts
      • or wherever Python is installed..
  3. Twisted and Zope
  4. Visual Studio Stuff
  5. !PyWin32
  6. Jinja2
    • easy_install jinja2 (as an admin)
  7. mock
    • easy_install mock (as an admin)
    • If you can "import mock" as an admin but not as an user, fix the DACL (access rights):
      • CACLS c:python26libsite-packagesmock-0.6.0-py2.6.egg /g buildbot:F
      • (assuming that's the path to the egg, and your non-admin user is 'buildbot')
  8. buildbot
  9. msysgit
  10. hack trial.bat
    • in C:Python26Scripts, create a file named "trial.bat" with only this line:
    • @"%~dp0..python" "%~dp0trial.py" %*
  11. hack trial.py in the same dir
    • in C:Python26Scripts, edit trial.py. The existing logic is broken, as it does not add '.' if getuid is not present. Change
      if hasattr(os, "getuid") and os.getuid() != 0:
          sys.path.insert(0, os.curdir)
      
      to
      #if hasattr(os, "getuid") and os.getuid() != 0:
      sys.path.insert(0, os.getcwd())
      
  12. If you want to have the user auto-login
    1. run "control userpasswords2"
    2. Uncheck "User must enter username and password to use this computer", click OK and enter the user credentials
    3. Add a shortcut in startup to 'c:slaveuild_slave.bat' with this script containing 'buildbot start'
    4. Reboot
  13. install sqlalchemy and sqlalchemy-migrate, checking that the buildbot user has permission to access them
  14. run "buildbot start"

Mac OS X 10.6 Setup

  1. Install MacPorts
    • This requires XCode, but X11 is not required
  2. Install the following ports
    • git-core
    • python24
    • python25
    • python26
    • python27
  3. (to be continued)