Host

The host machine for the Buildbot infrastructure is one of bear's systems, located in Germany. It is a 1.2GHz Celeron with 2GB of RAM and about 250GB of drive space. It is running Ubuntu 10.04.1. Its canonical hostname is ds0210.flosoft-servers.net.

Most Buildbot components are installed as buildbot, so root access is not required. Access to the buildbot account is controlled via SSH keys in ~/.ssh/authorized_keys2.

System-Level Installs

The following have been installed systemwide:

• nginx
• subversion
• git
• python-dev
• Amanda (from  http://www.zmanda.com/download-amanda.php)
• Apache 2.2 + libapache2-mod-rpaf + libapache2-mod-wsgi

Python

We are using the system Python (2.6.5), but all Python packages are installed in a virtualenv named ~/sandbox. This sandbox is automatically activated on login, identified by a (sandbox) prefix in the shell prompt.

Both Trac and Buildbot run out of the same virtualenv. This means we can easily back up and restore the Python environment, and need not worry about polluting other parts of the system.

Trac

Trac is hosted at  http://trac.buildbot.net, but I guess you knew that already. It is front-ended by nginx, backended by Apache. The trac "home" is ~/trac.

The following plugins are installed:

 pip install TracAccountManager
 pip install http://github.com/hvr/trac-git-plugin/tarball/master
 pip install TracGoogleAnalytics
 pip install http://trac-hacks.org/svn/tagsplugin/tags/0.6
 pip install http://trac-hacks.org/svn/navaddplugin/0.9
 git clone git://github.com/davglass/github-trac.git /tmp/ght && pip install /tmp/ght
 pip install http://github.com/hvr/trac-git-plugin/tarball/master
 pip install http://trac-hacks.org/svn/tocmacro/0.11

as well as the following distribution packages for display purposes

 pip install Pygments textile docutils

Trac currently authenticates with digested passwords, stored in ~/trac/conf.

Apache

Apache works as application server for trac, trac is running with WSGI mode. General installation instructions can be taken from  http://trac.edgewall.org/wiki/TracDev/AlternativeFrontends , section Alternative frontend: mod_wsgi. Apache listens port 8080 and nginx is proxying there.

Trac and Git

The github-trac plugin takes care of the work of redirecting 'Browse Source' and whatnot over to github, which is good. It also takes care of updating tickets via a "Trac" commit hook set up at github.

The tricky part is to get Trac to recognize git commits, display tooltips, search the commits, etc. Trac 0.12 has nice support built in for repositories, and the  GitPlugin handles these, but not without some work. See  TracRepositoryAdmin for the background.

The git mirror repository is in ~/trac/repos/buildbot.git. After setting up the config in the Admin screen with the name "Buildbot", I created the repository and resync'd it:

git clone --bare git://github.com/buildbot/buildbot.git ~/trac/repos/buildbot.git
trac-admin ~/trac repository resync Buildbot

I set up the repository's config to fetch directly:

[remote "origin"]
        url = git://github.com/buildbot/buildbot.git
        fetch = +refs/heads/*:refs/heads/*

and aliased the default repository to Buildbot:

trac-admin ~/trac repository alias '' "Buildbot"
trac-admin ~/trac repository set '' hidden true

This bare repo gets updated by the 'docs' builder in the metabuildbot, which subsequently calls

trac-admin ~/trac repository sync Buildbot

Trac's default sync-on-every-request functionality is disabled.

Metabuildbot

The metabuildbot ( http://buildbot.buildbot.net) runs on port 8010, and is front-ended by nginx. See MetaBuildbotConfig for more.

Nginx

Nginx serves several vhosts on port 80:

• buildbot.net
• trac.buildbot.net
• buildbot.buildbot.net

along with a few miscellaneous redirects.

Nginx proxies to port 8010 for the metabuildbot and port 3050 for Trac, applying caching for URIs rooted at /chrome.

Postfix

Postfix is installed as a a standard MTA with no special tweaks.

DNS

DNS for the buildbot.net domain is owned and controlled by Dustin. The domain is registered with godaddy, whom I will not do the favor of linking to, and served by  http://dyndns.com, who deserve the link love. Pretty much everything in the zone points to the same host.

Note that buildbot.org is currently registered and points to buildbot.net, but this will probably be allowed to lapse soon.

Backups

Backups are managed by Amanda, with the data sent to Amazon S3 on Zmanda's dime (and I do mean dime - S3 is *very* cheap for this quantity of data). The configuration is named bbnet and it backs up:

• /home/buildbot/metabuildbot (but not the builder basedirs)
• /home/buildbot/sandbox (although it could easily be re-created)
• /home/buildbot/trac
• /etc

Database Backups

From the buildbot crontab, at midnight local time the trac sqlite database is dumped as text to a file in the same directory. The nightly backups ignore the binary sqlite file and only back up the text version.

The metabuildbot's state database is not currently backed up, as the history is not particularly critical.

Static Web Files

A number of files are served from /home/buildbot/www/buildbot.net corresponding to  http://buildbot.net.

Static / Historical Documentation

The static "menu" pages and documentation for released versions of Buildbot are in the 'bbdocs' repository - see  http://github.com/buildbot/bbdocs. There is a clone of this repository at /home/buildbot/html/buildbot where you can 'git pull' as necessary. This data is all managed manually - new versions are added by hand and committed, and there is no automatic 'pull' going on on buildbot.net.

Note that there is a Python script, add-tracking.py, in the root of this branch. Run that script, feeding it the names of all of the HTML files, to add the Google Analytics bug so that we can track use of the documentation. All of this is usually handled as part of the release process (see CuttingReleases).

Latest Documentation

At every commit, the buildbot overwrites /home/buildbot/www/buildbot.net/buildbot/docs/latest with the latest generated texinfo and API docs. See the MetaBuildbotConfig for details about how this happens.

Coverage

The coverage information is also updated on every commit; it is at ~/www/buildbot.net/buildbot/coverage.

Scripts

There are several scripts in ~/bin:

• update-metabuildbot - this will update the metabuildbot's source repository to the latest on the master branch.

Log

• (?)08-May-2009: CoolCold? installed ngnix to cache non-logged-in users, and brought the load average down significantly
• (?)12-May-2009: Dustin noticed similar effects to  http://trac-hacks.org/ticket/3675 and applied  http://trac-hacks.org/attachment/ticket/3675/gitplugin.fix.patch
  • seems to have worked
• 01-Jul-2009 (warner): set up Git-based backup of buildbot.net trac
• 04-Jul-2009: Changed the nav config to point to the new buildbot (in /home/buildbot/trac-buildbot/conf/trac.ini)
• 04-Jul-2009: Dustin installed a buildbot on buildbot.net
• 31-Jul-2009: Dustin added a link to API docs generated by Thomas Vander Stichele
• 03-Jan-2010: Dustin freed up some disk space, installed logrotate, and installed the accountmanager trac plugin ( http://trac-hacks.org/wiki/AccountManagerPlugin); also set up /etc/crontab to blow away old buildmaster logs
• 03-Jan-2010: Dustin upgraded the buildmaster to 0.7.12rc1
• 16-Jan-2010: Dustin installed the github-trac plugin ( http://github.com/davglass/github-trac)
• 24-Jan-2010: Dustin edited the config for the metabuildbot receive-hook (in /etc/event.d/github_buildbot to point to /home/buildbot/github-buildbot-post-receive, then restarted it)
• 31-Jan-2010: On Nicolas's advice, added proxy_buffering off to the metabbot nginx config to enable streaming of logfiles
• 03-Feb-2010: Dustin removed /home/buildbot/apidocs and changed links to point to  http://djmitche.github.com/buildbot/docs/current/reference/. buildbot.net/apidocs is now a redirect to that URL (via nginx)
• 25-Feb-2010: Dustin, with help from verm__, switched to using a web form for authentication to try to limit the spam.
• 25-May-2010: Dustin installed  SpamFilter, but it doesn't seem to do anything.
• 27-May-2010: Dustin installed KeywordSuggestPlugin, which helps a lot, and reorganized the Development section.
• 10-Jul-2010: Dustin cleaned out a bunch of crap to free up disk space, added a bunch of config docs, and also moved the Buildbot docs to http://buildbot.net/buildbot/docs
• 10-Jul-2010: Dustin added backups, using Amanda and S3, to buildbot.net
• 22-Nov-2010: KeywordSuggestPlugin isn't working too well with Firefox betas. Dustin tried installing  TagsPlugin and  LoomingCloudsPlugin, which got a neat tag cloud, but didn't fix the autocomplete.
• 26-Nov-2010: Due to full disks, the HtDigest file was truncated sometime around the 16th. Dustin restored from backups and merged old and new accounts; also updated TracAccountManager as suggested by the author.
• 02-Dec-2010: Deleted 613 spams by 'shopp' - apparently I didn't read about them because mail was down when they were created (due to disk space). Also turned off "Trust authenticated users" in the spam detection plugin's config.
• 10-Dec-2010: Enabled Trac config to send mail to owners and reporters
• 20-Dec-2010 (approx): finished move from warner's linode to the new server
• 1-Jan-2011: Dustin disabled trac's sync-on-every-request and added a 'git fetch && trac-admin repository sync' to the metabuildbot; added docs here
• 13-Jan-2011: Dustin upgraded TracAccountManager to the latest version from svn, since apparently actually releasing software is "last-decade"
• 14-Jan-2011: Dustin edited templates/site.html to add hints to the account-registration and new-ticket pages, to help new users.
• 25-Jan-2011: Dustin started Trac after it got a bunch of database timeouts and exited.
• 1-Mar-2011: Dustin added ticket.href = /query to [mainnav] in trac.ini, pointing the "View Tickets" link to the query page. On Nicolas' advice.
• 4-Mar-2011: CoolCold? configured apache to run trac and reconfigured nginx to proxy on apache by Dustin's request
• 27-Mar-2011: Dustin added TocMacro to trac
• 1-Apr-2011: (#1903) Dustin added 'buildbot.net' to Postfix's $mydestinations so that 'nobody@…' would be deemed an acceptable email address.