wiki:BuildbotNetAdmin
Last modified 6 weeks ago

Table of Contents

  1. Admin Log
  2. Hosts
  3. System-Level Installs
  4. Python
  5. Trac
    1. Apache
    2. Trac and Git
  6. Metabuildbot
  7. Nginx
  8. Postfix
  9. DNS
  10. Backups
    1. Database Backups
  11. Static Web Files
    1. Static / Historical Documentation
    2. Latest Documentation
    3. Coverage
  12. Scripts
  13. New Buildbot.net
  14. VirtualHosts? and important URLs
  15. Log

Admin Log

This is the administration log for buildbot.net. Please record any changes you make here.

Hosts

The host machine for the Buildbot infrastructure is one of bear's systems, located in Germany. It is a 1.2GHz Celeron with 2GB of RAM and about 250GB of drive space. It is running Ubuntu 10.04.1. Its canonical hostname is ds0210.flosoft-servers.net.

We are also temporarily using a VM in Santa Clara, CA, US hosted by Mozilla, buildbot-trac.community.scl3.mozilla.com.

Most Buildbot components are installed as buildbot, so root access is not required. Access to the buildbot account is controlled via SSH keys in ~/.ssh/authorized_keys2.

System-Level Installs

The following have been installed systemwide:

Python

We are using the system Python (2.6.5), but all Python packages are installed in a virtualenv named ~/sandbox. This sandbox is automatically activated on login, identified by a (sandbox) prefix in the shell prompt.

Both Trac and Buildbot run out of the same virtualenv. This means we can easily back up and restore the Python environment, and need not worry about polluting other parts of the system.

Trac

Trac is hosted at  http://trac.buildbot.net, but I guess you knew that already. It is front-ended by nginx, backended by Apache. The trac "home" is ~/trac.

The following plugins are installed: 

 pip install TracAccountManager
 pip install http://github.com/hvr/trac-git-plugin/tarball/master
 pip install TracGoogleAnalytics
 pip install http://trac-hacks.org/svn/tagsplugin/tags/0.6
 pip install http://trac-hacks.org/svn/navaddplugin/0.9
 git clone git://github.com/davglass/github-trac.git /tmp/ght && pip install /tmp/ght
 pip install http://github.com/hvr/trac-git-plugin/tarball/master
 pip install http://trac-hacks.org/svn/tocmacro/0.11

as well as the following distribution packages for display purposes 

 pip install Pygments textile docutils

Trac currently authenticates with digested passwords, stored in ~/trac/conf.

Apache

Apache works as application server for trac, trac is running with WSGI mode. General installation instructions can be taken from  http://trac.edgewall.org/wiki/TracDev/AlternativeFrontends , section Alternative frontend: mod_wsgi. Apache listens port 8080 and nginx is proxying there.

Trac and Git

*NOTE* this is disabled for the moment, but is a good idea in general.

The github-trac plugin takes care of the work of redirecting 'Browse Source' and whatnot over to github, which is good. It also takes care of updating tickets via a "Trac" commit hook set up at github.

The tricky part is to get Trac to recognize git commits, display tooltips, search the commits, etc. Trac 0.12 has nice support built in for repositories, and the  GitPlugin handles these, but not without some work. See  TracRepositoryAdmin for the background.

The git mirror repository is in ~/trac/repos/buildbot.git. After setting up the config in the Admin screen with the name "Buildbot", I created the repository and resync'd it: 

git clone --bare git://github.com/buildbot/buildbot.git ~/trac/repos/buildbot.git
trac-admin ~/trac repository resync Buildbot

I set up the repository's config to fetch directly: 

[remote "origin"]
        url = git://github.com/buildbot/buildbot.git
        fetch = +refs/heads/*:refs/heads/*

and aliased the default repository to Buildbot: 

trac-admin ~/trac repository alias '' "Buildbot"
trac-admin ~/trac repository set '' hidden true

This bare repo gets updated by the 'docs' builder in the metabuildbot, which subsequently calls 

trac-admin ~/trac repository sync Buildbot

Trac's default sync-on-every-request functionality is disabled.

Metabuildbot

The metabuildbot ( http://buildbot.buildbot.net) runs on port 8010, and is front-ended by nginx. See MetaBuildbotConfig for more.

Nginx

Nginx serves several vhosts on port 80:

  • buildbot.net
  • trac.buildbot.net
  • buildbot.buildbot.net

along with a few miscellaneous redirects.

Nginx proxies to port 8010 for the metabuildbot and port 3050 for Trac, applying caching for URIs rooted at /chrome.

Postfix

Postfix is installed as a a standard MTA with no special tweaks.

DNS

DNS for the buildbot.net domain is owned and controlled by Dustin. The domain is registered with godaddy, whom I will not do the favor of linking to, and served by  http://dyndns.com, who deserve the link love. Pretty much everything in the zone points to the same host.

Note that buildbot.org is currently registered and points to buildbot.net, but this will probably be allowed to lapse soon.

Backups

Backups are managed by Amanda, with the data sent to Amazon S3 on Dustin's dime (and I do mean dime - S3 is *very* cheap for this quantity of data). The configuration is named bbnet and it backs up:

  • /home/buildbot/metabuildbot (but not the builder basedirs)
  • /home/buildbot/sandbox (although it could easily be re-created)
  • /home/buildbot/trac
  • /etc

Database Backups

From the buildbot crontab, at midnight local time the trac sqlite database is dumped as text to a file in the same directory. The nightly backups ignore the binary sqlite file and only back up the text version.

The metabuildbot's state database is not currently backed up, as the history is not particularly critical.

Static Web Files

A number of files are served from /home/buildbot/www/buildbot.net corresponding to  http://buildbot.net.

Static / Historical Documentation

The static "menu" pages and documentation for released versions of Buildbot are in the 'bbdocs' repository - see  http://github.com/buildbot/bbdocs. There is a clone of this repository at /home/buildbot/html/buildbot where you can 'git pull' as necessary. This data is all managed manually - new versions are added by hand and committed, and there is no automatic 'pull' going on on buildbot.net.

Note that there is a Python script, add-tracking.py, in the root of this branch. Run that script, feeding it the names of all of the HTML files, to add the Google Analytics bug so that we can track use of the documentation. All of this is usually handled as part of the release process (see CuttingReleases).

Latest Documentation

At every commit, the buildbot overwrites /home/buildbot/www/buildbot.net/buildbot/docs/latest with the latest generated texinfo and API docs. See the MetaBuildbotConfig for details about how this happens.

Coverage

The coverage information is also updated on every commit; it is at ~/www/buildbot.net/buildbot/coverage.

Scripts

There are several scripts in ~/bin:

  • update-metabuildbot - this will update the metabuildbot's source repository to the latest on the master branch.

New Buildbot.net

Source is from  https://github.com/buildbot/buildbot-website. This is installed on Ubuntu as follows:

VirtualHosts? and important URLs

Log

datewhowhat
2009-05-08CoolCold?installed ngnix to cache non-logged-in users, and brought the load average down significantly
2009-05-12Dustinnoticed similar effects to  http://trac-hacks.org/ticket/3675 and applied  http://trac-hacks.org/attachment/ticket/3675/gitplugin.fix.patch seems to have worked
2009-07-01warnerset up Git-based backup of buildbot.net trac
2009-07-04???Changed the nav config to point to the new buildbot (in /home/buildbot/trac-buildbot/conf/trac.ini)
2009-07-04Dustininstalled a buildbot on buildbot.net
2009-07-31Dustinadded a link to API docs generated by Thomas Vander Stichele
2010-01-03Dustinfreed up some disk space, installed logrotate, and installed the accountmanager trac plugin (  http://trac-hacks.org/wiki/AccountManagerPlugin); also set up /etc/crontab to blow away old buildmaster logs
2010-01-03Dustinupgraded the buildmaster to 0.7.12rc1
2010-01-16Dustininstalled the github-trac plugin (  http://github.com/davglass/github-trac)
2010-01-24Dustinedited the config for the metabuildbot receive-hook (in /etc/event.d/github_buildbot to point to /home/buildbot/github-buildbot-post-receive, then restarted it)
2010-01-31???On Nicolas's advice, added proxy_buffering off to the metabbot nginx config to enable streaming of logfiles
2010-02-03Dustinremoved /home/buildbot/apidocs and changed links to point to  http://djmitche.github.com/buildbot/docs/current/reference/. buildbot.net/apidocs is now a redirect to that URL (via nginx)
2010-02-25Dustinwith help from verm, switched to using a web form for authentication to try to limit the spam.
2010-05-25Dustininstalled SpamFilter?, but it doesn't seem to do anything.
2010-05-27Dustininstalled KeywordSuggestPlugin?, which helps a lot, and reorganized the Development section.
2010-07-10Dustinadded backups, using Amanda and S3, to buildbot.net
2010-07-10Dustincleaned out a bunch of crap to free up disk space, added a bunch of config docs, and also moved the Buildbot docs to http://buildbot.net/buildbot/docs
2010-11-22???KeywordSuggestPlugin? isn't working too well with Firefox betas. Dustin tried installing TagsPlugin? and LoomingCloudsPlugin?, which got a neat tag cloud, but didn't fix the autocomplete.
2010-11-26???Due to full disks, the HtDigest? file was truncated sometime around the 16th. Dustin restored from backups and merged old and new accounts; also updated TracAccountManager? as suggested by the author.
2010-12-02???Deleted 613 spams by 'shopp' - apparently I didn't read about them because mail was down when they were created (due to disk space). Also turned off "Trust authenticated users" in the spam detection plugin's config.
2010-12-10???Enabled Trac config to send mail to owners and reporters
2010-12-20???(approx): finished move from warner's linode to the new server
2011-01-01Dustindisabled trac's sync-on-every-request and added a 'git fetch && trac-admin repository sync' to the metabuildbot; added docs here
2011-01-13Dustinupgraded TracAccountManager? to the latest version from svn, since apparently actually releasing software is "last-decade"
2011-01-14Dustinedited templates/site.html to add hints to the account-registration and new-ticket pages, to help new users.
2011-01-25Dustinstarted Trac after it got a bunch of database timeouts and exited.
2011-03-01Dustinadded ticket.href = /query to [mainnav] in trac.ini, pointing the "View Tickets" link to the query page. On Nicolas' advice.
2011-03-04CoolCold?configured apache to run trac and reconfigured nginx to proxy on apache by Dustin's request
2011-03-27Dustinadded TocMacro? to trac
2011-04-01Dustin(#1903) added 'buildbot.net' to Postfix's $mydestinations so that 'nobody@…' would be deemed an acceptable email address.
2012-02-15vermenabled always_notify_updater in trac which will send emails to anyone that has updated or commented on a ticket. This was done by request.
2012-06-30bearsecurity updates and patches: touched nginx, apache, postgres, timezone data, compiler tools
2013-02-??dustindisabled trac git extension, removed repos -- too slow
2013-03-12dustinset up the new buildbot.net on buildbot-trac.community.scl3.mozilla.com
2013-04-??verminstalled Trac on buildbot-trac and transitioned to it
2013-04-??dustinmoved  http://buildbot.net to buildbot-trac