This document is badly out of date. Consider it a reference for how the systems looked, roughly, at the dawn of 2014.
The host machine for the Buildbot infrastructure is one of bear's systems, located in Germany. It is a 1.2GHz Celeron with 2GB of RAM and about 250GB of drive space. It is running Ubuntu 10.04.1. Its canonical hostname is ds0210.flosoft-servers.net.
We are also temporarily using a VM in Santa Clara, CA, US hosted by Mozilla, buildbot-trac.community.scl3.mozilla.com.
Most Buildbot components are installed as buildbot, so root access is not required. Access to the buildbot account is controlled via SSH keys in ~/.ssh/authorized_keys2.
The following have been installed systemwide:
- Amanda (from http://www.zmanda.com/download-amanda.php)
- Apache 2.2 + libapache2-mod-rpaf + libapache2-mod-wsgi
We are using the system Python (2.6.5), but all Python packages are installed in a virtualenv named ~/sandbox. This sandbox is automatically activated on login, identified by a (sandbox) prefix in the shell prompt.
Both Trac and Buildbot run out of the same virtualenv. This means we can easily back up and restore the Python environment, and need not worry about polluting other parts of the system.
Trac is hosted at http://trac.buildbot.net, but I guess you knew that already. It is front-ended by nginx, backended by Apache. The trac "home" is ~/trac.
The following plugins are installed:
pip install TracAccountManager pip install http://github.com/hvr/trac-git-plugin/tarball/master pip install TracGoogleAnalytics pip install http://trac-hacks.org/svn/tagsplugin/tags/0.6 pip install http://trac-hacks.org/svn/navaddplugin/0.9 git clone git://github.com/davglass/github-trac.git /tmp/ght && pip install /tmp/ght pip install http://github.com/hvr/trac-git-plugin/tarball/master pip install http://trac-hacks.org/svn/tocmacro/0.11
as well as the following distribution packages for display purposes
pip install Pygments textile docutils
Trac currently authenticates with digested passwords, stored in ~/trac/conf.
Apache works as application server for trac, trac is running with WSGI mode. General installation instructions can be taken from http://trac.edgewall.org/wiki/TracDev/AlternativeFrontends , section Alternative frontend: mod_wsgi. Apache listens port 8080 and nginx is proxying there.
Trac and Git
*NOTE* this is disabled for the moment, but is a good idea in general.
The github-trac plugin takes care of the work of redirecting 'Browse Source' and whatnot over to github, which is good. It also takes care of updating tickets via a "Trac" commit hook set up at github.
The tricky part is to get Trac to recognize git commits, display tooltips, search the commits, etc. Trac 0.12 has nice support built in for repositories, and the GitPlugin handles these, but not without some work. See TracRepositoryAdmin for the background.
The git mirror repository is in ~/trac/repos/buildbot.git. After setting up the config in the Admin screen with the name "Buildbot", I created the repository and resync'd it:
git clone --bare git://github.com/buildbot/buildbot.git ~/trac/repos/buildbot.git trac-admin ~/trac repository resync Buildbot
I set up the repository's config to fetch directly:
[remote "origin"] url = git://github.com/buildbot/buildbot.git fetch = +refs/heads/*:refs/heads/*
and aliased the default repository to Buildbot:
trac-admin ~/trac repository alias '' "Buildbot" trac-admin ~/trac repository set '' hidden true
This bare repo gets updated by the 'docs' builder in the metabuildbot, which subsequently calls
trac-admin ~/trac repository sync Buildbot
Trac's default sync-on-every-request functionality is disabled.
Nginx serves several vhosts on port 80:
along with a few miscellaneous redirects.
Nginx proxies to port 8010 for the metabuildbot and port 3050 for Trac, applying caching for URIs rooted at /chrome.
Postfix is installed as a a standard MTA with no special tweaks.
DNS for the buildbot.net domain is owned and controlled by Dustin. The domain is registered with godaddy, whom I will not do the favor of linking to, and served by http://dyndns.com, who deserve the link love. Pretty much everything in the zone points to the same host.
Note that buildbot.org is currently registered and points to buildbot.net, but this will probably be allowed to lapse soon.
Backups are managed by Amanda, with the data sent to Amazon S3 on Dustin's dime (and I do mean dime - S3 is *very* cheap for this quantity of data). The configuration is named bbnet and it backs up:
- /home/buildbot/metabuildbot (but not the builder basedirs)
- /home/buildbot/sandbox (although it could easily be re-created)
From the buildbot crontab, at midnight local time the trac sqlite database is dumped as text to a file in the same directory. The nightly backups ignore the binary sqlite file and only back up the text version.
The metabuildbot's state database is not currently backed up, as the history is not particularly critical.
Static Web Files
A number of files are served from /home/buildbot/www/buildbot.net corresponding to http://buildbot.net.
Static / Historical Documentation
The static "menu" pages and documentation for released versions of Buildbot are in the 'bbdocs' repository - see http://github.com/buildbot/bbdocs. There is a clone of this repository at /home/buildbot/html/buildbot where you can 'git pull' as necessary. This data is all managed manually - new versions are added by hand and committed, and there is no automatic 'pull' going on on buildbot.net.
Note that there is a Python script, add-tracking.py, in the root of this branch. Run that script, feeding it the names of all of the HTML files, to add the Google Analytics bug so that we can track use of the documentation. All of this is usually handled as part of the release process (see CuttingReleases).
At every commit, the buildbot overwrites /home/buildbot/www/buildbot.net/buildbot/docs/latest with the latest generated texinfo and API docs. See the MetaBuildbotConfig for details about how this happens.
The coverage information is also updated on every commit; it is at ~/www/buildbot.net/buildbot/coverage.
There are several scripts in ~/bin:
- update-metabuildbot - this will update the metabuildbot's source repository to the latest on the master branch.
This is set up in /var/lib/meetbot on ds0210, with a virtualenv in /var/lib/meetbot/virtualenv. The supybot install is from https://github.com/jamessan/Supybot and the meetbot install is from https://github.com/openstack-infra/meetbot (noting that setup.py doesn't actually work - see the readme). It runs as a dedicated user, meetbot. The bot is started by an @reboot task in that user's crontab. I know, great deployment process there, right?
The meetings are served from the http://meetings.buildbot.net vhost with a dead simple nginx config.
Source is from https://github.com/buildbot/buildbot-website. This is installed on Ubuntu as follows:
- set up a new site, buildbot-net, with root dir /var/www/buildbot.net/built
- installed a new nodejs and npm: http://askubuntu.com/questions/49390/how-do-i-install-the-latest-version-of-node-js
- clone buildbot-website to /var/www/buildbot.net
- cd there and run 'npm install', then 'grunt'.
VirtualHosts and important URLs
- http://buildbot.net (also www.*, buildbot.org) - just the documentation, but with a bunch of 301 redirects for old URLs
- http://buildbot.buildbot.net - the metabuildbot
- http://docs.buildbot.net - documentation
- http://trac.buildbot.net - Trac
|2014-01-29||dustin||set up supybot+meetbot on ds0210|
|2013-04-??||dustin||moved http://buildbot.net to buildbot-trac|
|2013-04-??||verm||installed Trac on buildbot-trac and transitioned to it|
|2013-03-12||dustin||set up the new buildbot.net on buildbot-trac.community.scl3.mozilla.com|
|2013-02-??||dustin||disabled trac git extension, removed repos -- too slow|
|2012-06-30||bear||security updates and patches: touched nginx, apache, postgres, timezone data, compiler tools|
|2012-02-15||verm||enabled always_notify_updater in trac which will send emails to anyone that has updated or commented on a ticket. This was done by request.|
|2011-04-01||Dustin||(#1903) added 'buildbot.net' to Postfix's $mydestinations so that 'nobody@…' would be deemed an acceptable email address.|
|2011-03-27||Dustin||added TocMacro? to trac|
|2011-03-04||CoolCold?||configured apache to run trac and reconfigured nginx to proxy on apache by Dustin's request|
|2011-03-01||Dustin||added ticket.href = /query to [mainnav] in trac.ini, pointing the "View Tickets" link to the query page. On Nicolas' advice.|
|2011-01-25||Dustin||started Trac after it got a bunch of database timeouts and exited.|
|2011-01-14||Dustin||edited templates/site.html to add hints to the account-registration and new-ticket pages, to help new users.|
|2011-01-13||Dustin||upgraded TracAccountManager? to the latest version from svn, since apparently actually releasing software is "last-decade"|
|2011-01-01||Dustin||disabled trac's sync-on-every-request and added a 'git fetch && trac-admin repository sync' to the metabuildbot; added docs here|
|2010-12-20||???||(approx): finished move from warner's linode to the new server|
|2010-12-10||???||Enabled Trac config to send mail to owners and reporters|
|2010-12-02||???||Deleted 613 spams by 'shopp' - apparently I didn't read about them because mail was down when they were created (due to disk space). Also turned off "Trust authenticated users" in the spam detection plugin's config.|
|2010-11-26||???||Due to full disks, the HtDigest? file was truncated sometime around the 16th. Dustin restored from backups and merged old and new accounts; also updated TracAccountManager? as suggested by the author.|
|2010-11-22||???||KeywordSuggestPlugin? isn't working too well with Firefox betas. Dustin tried installing TagsPlugin? and LoomingCloudsPlugin?, which got a neat tag cloud, but didn't fix the autocomplete.|
|2010-07-10||Dustin||cleaned out a bunch of crap to free up disk space, added a bunch of config docs, and also moved the Buildbot docs to http://buildbot.net/buildbot/docs|
|2010-07-10||Dustin||added backups, using Amanda and S3, to buildbot.net|
|2010-05-27||Dustin||installed KeywordSuggestPlugin?, which helps a lot, and reorganized the Development section.|
|2010-05-25||Dustin||installed SpamFilter?, but it doesn't seem to do anything.|
|2010-02-25||Dustin||with help from verm, switched to using a web form for authentication to try to limit the spam.|
|2010-02-03||Dustin||removed /home/buildbot/apidocs and changed links to point to http://djmitche.github.com/buildbot/docs/current/reference/. buildbot.net/apidocs is now a redirect to that URL (via nginx)|
|2010-01-31||???||On Nicolas's advice, added proxy_buffering off to the metabbot nginx config to enable streaming of logfiles|
|2010-01-24||Dustin||edited the config for the metabuildbot receive-hook (in /etc/event.d/github_buildbot to point to /home/buildbot/github-buildbot-post-receive, then restarted it)|
|2010-01-16||Dustin||installed the github-trac plugin ( http://github.com/davglass/github-trac)|
|2010-01-03||Dustin||upgraded the buildmaster to 0.7.12rc1|
|2010-01-03||Dustin||freed up some disk space, installed logrotate, and installed the accountmanager trac plugin ( http://trac-hacks.org/wiki/AccountManagerPlugin); also set up /etc/crontab to blow away old buildmaster logs|
|2009-07-31||Dustin||added a link to API docs generated by Thomas Vander Stichele|
|2009-07-04||Dustin||installed a buildbot on buildbot.net|
|2009-07-04||???||Changed the nav config to point to the new buildbot (in /home/buildbot/trac-buildbot/conf/trac.ini)|
|2009-07-01||warner||set up Git-based backup of buildbot.net trac|
|2009-05-12||Dustin||noticed similar effects to http://trac-hacks.org/ticket/3675 and applied http://trac-hacks.org/attachment/ticket/3675/gitplugin.fix.patch seems to have worked|
|2009-05-08||CoolCold?||installed ngnix to cache non-logged-in users, and brought the load average down significantly|