test real RemoteCommands and FakeRemoteCommands with the same tests

[tom.prince]

Once we have pluggable protocols for the master-slave communication, we should have a fake-slave protocols for testing against, rather than monkey-patching fake RemoteCommand? instances.

[dustin]

I'm not totally convinced this is worthwhile -- right now, the tests verify that the steps correctly interact with the RemoteCommand interface. If we are also testing the RemoteCommand interface at the same time, then the tests are no longer unit tests.

We should definitely test the RemoteCommand implementation against a fake slave - we do a little bit of that now, but it can be much more thorough with a pluggable backend.

[tom.prince]

Well, we need to test the tests actually *do* test the RemoteCommand? interface. I just discovered that eb7bf214767f95304b5ade9938fe369320c17eef doesn't actually work, since RemoteShellCommand? doesn't actually match the interface we are testing against there.

[dustin]

Summary adjusted with what I think is a better solution here -- test that the fakes and the real implementations pass the same tests -  http://pyvideo.org/video/629/stop-mocking-start-testing

From discussion in irc, though, it sounds like this case was a little more tricky -- an invalid function keyword argument was added to FakeRemoteCommand that was not present in RemoteCommand. I think there are three ways to uncover problems like this, and we should aim to do all three:

• test fakes and real objects with the same tests
  • use introspection to make assertions about function/method arguments
• once this is in place and known, code review should look for parallelism in changes to real and fake objects
• light use of integration testing as a fallback

I'm not a big fan of heavy-duty integration testing, as it's *very* difficult to hit every possibility. In this case, a few integration tests running e.g., Git steps against a fake slave may not have triggered this particular error.