This document is badly out of date. Consider it a reference for how the systems looked, roughly, at the dawn of 2014.


The host machine for the Buildbot infrastructure is one of bear's systems, located in Germany. It is a 1.2GHz Celeron with 2GB of RAM and about 250GB of drive space. It is running Ubuntu 10.04.1. Its canonical hostname is

We are also temporarily using a VM in Santa Clara, CA, US hosted by Mozilla,

Most Buildbot components are installed as buildbot, so root access is not required. Access to the buildbot account is controlled via SSH keys in ~/.ssh/authorized_keys2.

System-Level Installs

The following have been installed systemwide:


We are using the system Python (2.6.5), but all Python packages are installed in a virtualenv named ~/sandbox. This sandbox is automatically activated on login, identified by a (sandbox) prefix in the shell prompt.

Both Trac and Buildbot run out of the same virtualenv. This means we can easily back up and restore the Python environment, and need not worry about polluting other parts of the system.


Trac is hosted at, but I guess you knew that already. It is front-ended by nginx, backended by Apache. The trac "home" is ~/trac.

The following plugins are installed:

 pip install TracAccountManager
 pip install
 pip install TracGoogleAnalytics
 pip install
 pip install
 git clone git:// /tmp/ght && pip install /tmp/ght
 pip install
 pip install

as well as the following distribution packages for display purposes

 pip install Pygments textile docutils

Trac currently authenticates with digested passwords, stored in ~/trac/conf.


Apache works as application server for trac, trac is running with WSGI mode. General installation instructions can be taken from , section Alternative frontend: mod_wsgi. Apache listens port 8080 and nginx is proxying there.

Trac and Git

*NOTE* this is disabled for the moment, but is a good idea in general.

The github-trac plugin takes care of the work of redirecting 'Browse Source' and whatnot over to github, which is good. It also takes care of updating tickets via a "Trac" commit hook set up at github.

The tricky part is to get Trac to recognize git commits, display tooltips, search the commits, etc. Trac 0.12 has nice support built in for repositories, and the GitPlugin handles these, but not without some work. See TracRepositoryAdmin for the background.

The git mirror repository is in ~/trac/repos/buildbot.git. After setting up the config in the Admin screen with the name "Buildbot", I created the repository and resync'd it:

git clone --bare git:// ~/trac/repos/buildbot.git
trac-admin ~/trac repository resync Buildbot

I set up the repository's config to fetch directly:

[remote "origin"]
        url = git://
        fetch = +refs/heads/*:refs/heads/*

and aliased the default repository to Buildbot:

trac-admin ~/trac repository alias '' "Buildbot"
trac-admin ~/trac repository set '' hidden true

This bare repo gets updated by the 'docs' builder in the metabuildbot, which subsequently calls

trac-admin ~/trac repository sync Buildbot

Trac's default sync-on-every-request functionality is disabled.


The metabuildbot ( runs on port 8010, and is front-ended by nginx. See MetaBuildbotConfig for more.


Nginx serves several vhosts on port 80:


along with a few miscellaneous redirects.

Nginx proxies to port 8010 for the metabuildbot and port 3050 for Trac, applying caching for URIs rooted at /chrome.


Postfix is installed as a a standard MTA with no special tweaks.


DNS for the domain is owned and controlled by Dustin. The domain is registered with godaddy, whom I will not do the favor of linking to, and served by, who deserve the link love. Pretty much everything in the zone points to the same host.

Note that is currently registered and points to, but this will probably be allowed to lapse soon.


Backups are managed by Amanda, with the data sent to Amazon S3 on Dustin's dime (and I do mean dime - S3 is *very* cheap for this quantity of data). The configuration is named bbnet and it backs up:

  • /home/buildbot/metabuildbot (but not the builder basedirs)
  • /home/buildbot/sandbox (although it could easily be re-created)
  • /home/buildbot/trac
  • /etc

Database Backups

From the buildbot crontab, at midnight local time the trac sqlite database is dumped as text to a file in the same directory. The nightly backups ignore the binary sqlite file and only back up the text version.

The metabuildbot's state database is not currently backed up, as the history is not particularly critical.

Static Web Files

A number of files are served from /home/buildbot/www/ corresponding to

Static / Historical Documentation

The static "menu" pages and documentation for released versions of Buildbot are in the 'bbdocs' repository - see There is a clone of this repository at /home/buildbot/html/buildbot where you can 'git pull' as necessary. This data is all managed manually - new versions are added by hand and committed, and there is no automatic 'pull' going on on

Note that there is a Python script,, in the root of this branch. Run that script, feeding it the names of all of the HTML files, to add the Google Analytics bug so that we can track use of the documentation. All of this is usually handled as part of the release process (see CuttingReleases).

Latest Documentation

At every commit, the buildbot overwrites /home/buildbot/www/ with the latest generated texinfo and API docs. See the MetaBuildbotConfig for details about how this happens.


The coverage information is also updated on every commit; it is at ~/www/


There are several scripts in ~/bin:

  • update-metabuildbot - this will update the metabuildbot's source repository to the latest on the master branch.

MeetBot (bb-meeting)

This is set up in /var/lib/meetbot on ds0210, with a virtualenv in /var/lib/meetbot/virtualenv. The supybot install is from and the meetbot install is from (noting that doesn't actually work - see the readme). It runs as a dedicated user, meetbot. The bot is started by an @reboot task in that user's crontab. I know, great deployment process there, right?

The meetings are served from the vhost with a dead simple nginx config.


Source is from This is installed on Ubuntu as follows:

VirtualHosts and important URLs


2014-01-29dustinset up supybot+meetbot on ds0210
2013-04-??dustinmoved to buildbot-trac
2013-04-??verminstalled Trac on buildbot-trac and transitioned to it
2013-03-12dustinset up the new on
2013-02-??dustindisabled trac git extension, removed repos -- too slow
2012-06-30bearsecurity updates and patches: touched nginx, apache, postgres, timezone data, compiler tools
2012-02-15vermenabled always_notify_updater in trac which will send emails to anyone that has updated or commented on a ticket. This was done by request.
2011-04-01Dustin(#1903) added '' to Postfix's $mydestinations so that 'nobody@…' would be deemed an acceptable email address.
2011-03-27Dustinadded TocMacro? to trac
2011-03-04CoolCold?configured apache to run trac and reconfigured nginx to proxy on apache by Dustin's request
2011-03-01Dustinadded ticket.href = /query to [mainnav] in trac.ini, pointing the "View Tickets" link to the query page. On Nicolas' advice.
2011-01-25Dustinstarted Trac after it got a bunch of database timeouts and exited.
2011-01-14Dustinedited templates/site.html to add hints to the account-registration and new-ticket pages, to help new users.
2011-01-13Dustinupgraded TracAccountManager? to the latest version from svn, since apparently actually releasing software is "last-decade"
2011-01-01Dustindisabled trac's sync-on-every-request and added a 'git fetch && trac-admin repository sync' to the metabuildbot; added docs here
2010-12-20???(approx): finished move from warner's linode to the new server
2010-12-10???Enabled Trac config to send mail to owners and reporters
2010-12-02???Deleted 613 spams by 'shopp' - apparently I didn't read about them because mail was down when they were created (due to disk space). Also turned off "Trust authenticated users" in the spam detection plugin's config.
2010-11-26???Due to full disks, the HtDigest? file was truncated sometime around the 16th. Dustin restored from backups and merged old and new accounts; also updated TracAccountManager? as suggested by the author.
2010-11-22???KeywordSuggestPlugin? isn't working too well with Firefox betas. Dustin tried installing TagsPlugin? and LoomingCloudsPlugin?, which got a neat tag cloud, but didn't fix the autocomplete.
2010-07-10Dustincleaned out a bunch of crap to free up disk space, added a bunch of config docs, and also moved the Buildbot docs to
2010-07-10Dustinadded backups, using Amanda and S3, to
2010-05-27Dustininstalled KeywordSuggestPlugin?, which helps a lot, and reorganized the Development section.
2010-05-25Dustininstalled SpamFilter?, but it doesn't seem to do anything.
2010-02-25Dustinwith help from verm, switched to using a web form for authentication to try to limit the spam.
2010-02-03Dustinremoved /home/buildbot/apidocs and changed links to point to is now a redirect to that URL (via nginx)
2010-01-31???On Nicolas's advice, added proxy_buffering off to the metabbot nginx config to enable streaming of logfiles
2010-01-24Dustinedited the config for the metabuildbot receive-hook (in /etc/event.d/github_buildbot to point to /home/buildbot/github-buildbot-post-receive, then restarted it)
2010-01-16Dustininstalled the github-trac plugin (
2010-01-03Dustinupgraded the buildmaster to 0.7.12rc1
2010-01-03Dustinfreed up some disk space, installed logrotate, and installed the accountmanager trac plugin (; also set up /etc/crontab to blow away old buildmaster logs
2009-07-31Dustinadded a link to API docs generated by Thomas Vander Stichele
2009-07-04Dustininstalled a buildbot on
2009-07-04???Changed the nav config to point to the new buildbot (in /home/buildbot/trac-buildbot/conf/trac.ini)
2009-07-01warnerset up Git-based backup of trac
2009-05-12Dustinnoticed similar effects to and applied seems to have worked
2009-05-08CoolCold?installed ngnix to cache non-logged-in users, and brought the load average down significantly
Last modified 4 years ago Last modified on Jan 4, 2017, 1:33:55 AM