Table of Contents
This is the administration log for buildbot.net. Please record any changes you make here.
The host machine for the Buildbot infrastructure is one of bear's systems, located in Germany. It is a 1.2GHz Celeron with 2GB of RAM and about 250GB of drive space. It is running Ubuntu 10.04.1. Its canonical hostname is ds0210.flosoft-servers.net.
We are also temporarily using a VM in Santa Clara, CA, US hosted by Mozilla, buildbot-trac.community.scl3.mozilla.com.
Most Buildbot components are installed as buildbot, so root access is not required. Access to the buildbot account is controlled via SSH keys in ~/.ssh/authorized_keys2.
The following have been installed systemwide:
- Amanda (from http://www.zmanda.com/download-amanda.php)
- Apache 2.2 + libapache2-mod-rpaf + libapache2-mod-wsgi
We are using the system Python (2.6.5), but all Python packages are installed in a virtualenv named ~/sandbox. This sandbox is automatically activated on login, identified by a (sandbox) prefix in the shell prompt.
Both Trac and Buildbot run out of the same virtualenv. This means we can easily back up and restore the Python environment, and need not worry about polluting other parts of the system.
Trac is hosted at http://trac.buildbot.net, but I guess you knew that already. It is front-ended by nginx, backended by Apache. The trac "home" is ~/trac.
The following plugins are installed:
pip install TracAccountManager pip install http://github.com/hvr/trac-git-plugin/tarball/master pip install TracGoogleAnalytics pip install http://trac-hacks.org/svn/tagsplugin/tags/0.6 pip install http://trac-hacks.org/svn/navaddplugin/0.9 git clone git://github.com/davglass/github-trac.git /tmp/ght && pip install /tmp/ght pip install http://github.com/hvr/trac-git-plugin/tarball/master pip install http://trac-hacks.org/svn/tocmacro/0.11
as well as the following distribution packages for display purposes
pip install Pygments textile docutils
Trac currently authenticates with digested passwords, stored in ~/trac/conf.
Apache works as application server for trac, trac is running with WSGI mode. General installation instructions can be taken from http://trac.edgewall.org/wiki/TracDev/AlternativeFrontends , section Alternative frontend: mod_wsgi. Apache listens port 8080 and nginx is proxying there.
Trac and Git
*NOTE* this is disabled for the moment, but is a good idea in general.
The github-trac plugin takes care of the work of redirecting 'Browse Source' and whatnot over to github, which is good. It also takes care of updating tickets via a "Trac" commit hook set up at github.
The tricky part is to get Trac to recognize git commits, display tooltips, search the commits, etc. Trac 0.12 has nice support built in for repositories, and the GitPlugin handles these, but not without some work. See TracRepositoryAdmin for the background.
The git mirror repository is in ~/trac/repos/buildbot.git. After setting up the config in the Admin screen with the name "Buildbot", I created the repository and resync'd it:
git clone --bare git://github.com/buildbot/buildbot.git ~/trac/repos/buildbot.git trac-admin ~/trac repository resync Buildbot
I set up the repository's config to fetch directly:
[remote "origin"] url = git://github.com/buildbot/buildbot.git fetch = +refs/heads/*:refs/heads/*
and aliased the default repository to Buildbot:
trac-admin ~/trac repository alias '' "Buildbot" trac-admin ~/trac repository set '' hidden true
This bare repo gets updated by the 'docs' builder in the metabuildbot, which subsequently calls
trac-admin ~/trac repository sync Buildbot
Trac's default sync-on-every-request functionality is disabled.
Nginx serves several vhosts on port 80:
along with a few miscellaneous redirects.
Nginx proxies to port 8010 for the metabuildbot and port 3050 for Trac, applying caching for URIs rooted at /chrome.
Postfix is installed as a a standard MTA with no special tweaks.
DNS for the buildbot.net domain is owned and controlled by Dustin. The domain is registered with godaddy, whom I will not do the favor of linking to, and served by http://dyndns.com, who deserve the link love. Pretty much everything in the zone points to the same host.
Note that buildbot.org is currently registered and points to buildbot.net, but this will probably be allowed to lapse soon.
Backups are managed by Amanda, with the data sent to Amazon S3 on Dustin's dime (and I do mean dime - S3 is *very* cheap for this quantity of data). The configuration is named bbnet and it backs up:
- /home/buildbot/metabuildbot (but not the builder basedirs)
- /home/buildbot/sandbox (although it could easily be re-created)
From the buildbot crontab, at midnight local time the trac sqlite database is dumped as text to a file in the same directory. The nightly backups ignore the binary sqlite file and only back up the text version.
The metabuildbot's state database is not currently backed up, as the history is not particularly critical.
Static Web Files
A number of files are served from /home/buildbot/www/buildbot.net corresponding to http://buildbot.net.
Static / Historical Documentation
The static "menu" pages and documentation for released versions of Buildbot are in the 'bbdocs' repository - see http://github.com/buildbot/bbdocs. There is a clone of this repository at /home/buildbot/html/buildbot where you can 'git pull' as necessary. This data is all managed manually - new versions are added by hand and committed, and there is no automatic 'pull' going on on buildbot.net.
Note that there is a Python script, add-tracking.py, in the root of this branch. Run that script, feeding it the names of all of the HTML files, to add the Google Analytics bug so that we can track use of the documentation. All of this is usually handled as part of the release process (see CuttingReleases).
At every commit, the buildbot overwrites /home/buildbot/www/buildbot.net/buildbot/docs/latest with the latest generated texinfo and API docs. See the MetaBuildbotConfig for details about how this happens.
The coverage information is also updated on every commit; it is at ~/www/buildbot.net/buildbot/coverage.
There are several scripts in ~/bin:
- update-metabuildbot - this will update the metabuildbot's source repository to the latest on the master branch.
Source is from https://github.com/buildbot/buildbot-website. This is installed on Ubuntu as follows:
- set up a new site, buildbot-net, with root dir /var/www/buildbot.net/built
- installed a new nodejs and npm: http://askubuntu.com/questions/49390/how-do-i-install-the-latest-version-of-node-js
- clone buildbot-website to /var/www/buildbot.net
- cd there and run 'npm install', then 'grunt'.
VirtualHosts? and important URLs
- http://buildbot.net (also www.*, buildbot.org) - just the documentation, but with a bunch of 301 redirects for old URLs
- http://buildbot.buildbot.net - the metabuildbot
- http://docs.buildbot.net - documentation
- http://trac.buildbot.net - Trac
|2009-05-08||CoolCold?||installed ngnix to cache non-logged-in users, and brought the load average down significantly|
|2009-05-12||Dustin||noticed similar effects to http://trac-hacks.org/ticket/3675 and applied http://trac-hacks.org/attachment/ticket/3675/gitplugin.fix.patch seems to have worked|
|2009-07-01||warner||set up Git-based backup of buildbot.net trac|
|2009-07-04||???||Changed the nav config to point to the new buildbot (in /home/buildbot/trac-buildbot/conf/trac.ini)|
|2009-07-04||Dustin||installed a buildbot on buildbot.net|
|2009-07-31||Dustin||added a link to API docs generated by Thomas Vander Stichele|
|2010-01-03||Dustin||freed up some disk space, installed logrotate, and installed the accountmanager trac plugin ( http://trac-hacks.org/wiki/AccountManagerPlugin); also set up /etc/crontab to blow away old buildmaster logs|
|2010-01-03||Dustin||upgraded the buildmaster to 0.7.12rc1|
|2010-01-16||Dustin||installed the github-trac plugin ( http://github.com/davglass/github-trac)|
|2010-01-24||Dustin||edited the config for the metabuildbot receive-hook (in /etc/event.d/github_buildbot to point to /home/buildbot/github-buildbot-post-receive, then restarted it)|
|2010-01-31||???||On Nicolas's advice, added proxy_buffering off to the metabbot nginx config to enable streaming of logfiles|
|2010-02-03||Dustin||removed /home/buildbot/apidocs and changed links to point to http://djmitche.github.com/buildbot/docs/current/reference/. buildbot.net/apidocs is now a redirect to that URL (via nginx)|
|2010-02-25||Dustin||with help from verm, switched to using a web form for authentication to try to limit the spam.|
|2010-05-25||Dustin||installed SpamFilter?, but it doesn't seem to do anything.|
|2010-05-27||Dustin||installed KeywordSuggestPlugin?, which helps a lot, and reorganized the Development section.|
|2010-07-10||Dustin||added backups, using Amanda and S3, to buildbot.net|
|2010-07-10||Dustin||cleaned out a bunch of crap to free up disk space, added a bunch of config docs, and also moved the Buildbot docs to http://buildbot.net/buildbot/docs|
|2010-11-22||???||KeywordSuggestPlugin? isn't working too well with Firefox betas. Dustin tried installing TagsPlugin? and LoomingCloudsPlugin?, which got a neat tag cloud, but didn't fix the autocomplete.|
|2010-11-26||???||Due to full disks, the HtDigest? file was truncated sometime around the 16th. Dustin restored from backups and merged old and new accounts; also updated TracAccountManager? as suggested by the author.|
|2010-12-02||???||Deleted 613 spams by 'shopp' - apparently I didn't read about them because mail was down when they were created (due to disk space). Also turned off "Trust authenticated users" in the spam detection plugin's config.|
|2010-12-10||???||Enabled Trac config to send mail to owners and reporters|
|2010-12-20||???||(approx): finished move from warner's linode to the new server|
|2011-01-01||Dustin||disabled trac's sync-on-every-request and added a 'git fetch && trac-admin repository sync' to the metabuildbot; added docs here|
|2011-01-13||Dustin||upgraded TracAccountManager? to the latest version from svn, since apparently actually releasing software is "last-decade"|
|2011-01-14||Dustin||edited templates/site.html to add hints to the account-registration and new-ticket pages, to help new users.|
|2011-01-25||Dustin||started Trac after it got a bunch of database timeouts and exited.|
|2011-03-01||Dustin||added ticket.href = /query to [mainnav] in trac.ini, pointing the "View Tickets" link to the query page. On Nicolas' advice.|
|2011-03-04||CoolCold?||configured apache to run trac and reconfigured nginx to proxy on apache by Dustin's request|
|2011-03-27||Dustin||added TocMacro? to trac|
|2011-04-01||Dustin||(#1903) added 'buildbot.net' to Postfix's $mydestinations so that 'nobody@…' would be deemed an acceptable email address.|
|2012-02-15||verm||enabled always_notify_updater in trac which will send emails to anyone that has updated or commented on a ticket. This was done by request.|
|2012-06-30||bear||security updates and patches: touched nginx, apache, postgres, timezone data, compiler tools|
|2013-02-??||dustin||disabled trac git extension, removed repos -- too slow|
|2013-03-12||dustin||set up the new buildbot.net on buildbot-trac.community.scl3.mozilla.com|
|2013-04-??||verm||installed Trac on buildbot-trac and transitioned to it|
|2013-04-??||dustin||moved http://buildbot.net to buildbot-trac|