| 1 | | Buildbot has a lot of tests, but very few of them are well-isolated or reliable, so they may not make the best examples. They are now in the {{{buildbot.broken_test}}} package. |
| 2 | | |
| 3 | | In general, we are trying to ensure that new tests are ''good''. These new, good tests are in {{{buildbot.test}}}. |
| 4 | | |
| 5 | | So what makes a good test? |
| 6 | | |
| 7 | | = Independent of Time = |
| 8 | | Tests that depend on wall time will fail. As a bonus, they run very slowly. Do not use {{{reactor.callLater}}} to wait "long enough" for something to happen. |
| 9 | | |
| 10 | | For testing things that themselves depend on time, consider using [http://twistedmatrix.com/documents/current/api/twisted.internet.task.Clock.html t.i.t.Clock]. This may mean passing a clock instance to the code under test, and propagating that instance as necessary to ensure that all of the code using {{{callLater}}} uses it. |
| 11 | | |
| 12 | | For testing things that do not depend on time, but for which you cannot detect the "end" of an operation: add a way to detect the end of the operation! |
| 13 | | |
| 14 | | = Clean Code = |
| 15 | | Make your tests readable. This is no place to skimp on comments! Others will attempt to learn about the expected behavior of your class by reading the tests. As a side note, if you use a Deferred chain in your test, write the callbacks as nested functions, rather than using object methods with funny names: |
| 16 | | |
| 17 | | {{{ |
| 18 | | def testSomething(self): |
| 19 | | d = doThisFirst() |
| 20 | | def andThisNext(res): |
| 21 | | pass # ... |
| 22 | | d.addCallback(andThisNext) |
| 23 | | return d |
| 24 | | }}} |
| 25 | | |
| 26 | | This isolates the entire test into one indented block. It is OK to add methods for common functionality, but give them real names and explain in detail what they do. |
| 27 | | |
| 28 | | = Assert Only One Thing = |
| 29 | | Each test should have a single assertion. This may require a little bit of work to get several related pieces of information into a single Python object for comparison. The problem with multiple assertions is that, if the first assertion fails, the remainder are not tested. If you need to make two unrelated assertions, you should be running two tests. |
| 30 | | |
| 31 | | = Use Mocks and Stubs = |
| 32 | | Mocks assert that they are called correctly. Stubs provide a predictable base on which to run the code under test. See http://en.wikipedia.org/wiki/Mock_object and http://en.wikipedia.org/wiki/Method_stub. |
| 33 | | |
| 34 | | = Small Tests = |
| 35 | | The shorter each test is, the better. Test as little code as possible in each test. |
| 36 | | |
| 37 | | It is fine, and in fact encouraged, to write the code under test in such a way as to facilitate this. As an illustrative example, if you are testing a new {{{Step}}} subclass, but your tests require instantiating a !BuildMaster, you're probably doing something wrong! (Note that this rule is almost universally violated in the existing buildbot tests). |
| 38 | | |
| 39 | | = Isolation = |
| 40 | | Each test should be maximally independent of other tests. Do not leave files laying around after your test has finished, and do not assume that some other test has run beforehand. It's fine to use caching techniques to avoi repeated, lengthy setup times. |
| 41 | | |
| 42 | | = Be Correct = |
| 43 | | Tests should be as robust as possible, which at a basic level means using the available frameworks correctly. All deferreds should have callbacks and be chained properly. Error conditions should be checked properly. Race conditions should not exist (see "Independent of Time"). |
| 44 | | |
| 45 | | = Be Helpful = |
| 46 | | When the test fails, it should produce output that is helpful to the person chasing it down. This is particularly important when the tests are run remotely, in which case the person chasing down the bug does not have access to the system on which the test fails. A test which fails sporadically with no more information than "AssertionFailed" is a prime candidate for deletion if the error isn't obvious. Making the error obvious also includes adding comments describing the ways a test might fail. |
| 47 | | |
| 48 | | = Good Name = |
| 49 | | Your test module should be named after the package or class it tests, replacing . with _ and omitting the {{{buildbot_}}}. For example, {{{test_status_web_authz_Authz.py}}} tests the {{{Authz}}} class in {{{buildbot/status/web/authz.py}}}. Modules with only one class, or a few trivial classes, can be tested in a single test module. For more complex situations, prefer to use multiple test modules. |
| 50 | | |
| 51 | | Test method names should follow the pattern {{{test_METHOD_CONDITION}}} where METHOD is the method being tested, and CONDITION is the condition under which it's tested. Since we can't always test a single method, this is not a hard-and-fast rule. |
| 52 | | |
| 53 | | = Mixins = |
| 54 | | Do not define {{{setUp}}} and {{{tearDown}}} directly in a mixin. This is the path to madness. Instead, define a {{{mixinNameSetUp}}} and {{{mixinNameTearDown}}}, and call them from the subclass's {{{setUp}}} and {{{tearDown}}}. This makes it perfectly clear what is being set up and torn down from a simple analysis of the test case. |
| | 1 | Page moved into the Buildbot documentation: |
| | 2 | http://djmitche.github.com/buildbot/docs/latest/Buildbot-Tests.html |
![[Buildbot Logo]](/chrome/site/header-text-transparent.png)