Wednesday, February 6, 2013

Mock Assurances

I recently tried the mock library; it's quite useful, and in general using it was a pleasant experience... until things turned scary. While refactoring some code and corresponding tests I hit a point where a test should have been failing, and yet was nonetheless passing. A little investigation led me to the problem, and that's when I got really nervous.

Here's a rather silly example of two unit tests using mock.
import unittest
import mock

class C:
    def function(self, x):

class Tests(unittest.TestCase):
    def test_positive(self):
        C2 = mock.Mock(spec=C)
        obj = C2()

    def test_negative(self):
        C2 = mock.Mock(spec=C)
        obj = C2()

if __name__ == '__main__':
One would expect test_negative to fail, but in fact both tests pass:
$ python 
Ran 2 tests in 0.001s


If you've used mock before, you probably know what's going on. The mock library creates new attributes on demand if they don't exist. Thus:
>>> import mock
>>> obj = mock.Mock()
>>> obj.assert_called_once_with
<bound method Mock.assert_called_once_with of <mock.Mock object at 0x7f6e1c18a990>>
>>> obj.assert_not_called
<Mock name='mock.assert_not_called' id='140110894509008'>
Once again, oops. I had invented a new assertion method assert_not_called which doesn't actually exist, and mock had happily created a new object for me. My test was completely broken. My mistake, and therefore my fault. And in fact the mock documentation does mention the possibility of this happening, deep within the bowels of the API documentation: "Because mocks auto-create attributes on demand, and allow you to call them with arbitrary arguments, if you misspell one of these assert methods then your assertion is gone." An appropriate fix is provided, the mock.create_autospec API. It would have been better to include this suggestion in the intro documentation; better yet would be preventing the problem in the first place.

By their very nature, test assertions do nothing silently in the expected case. It's thus quite dangerous to have a library specifically intended for testing where typos create calls that are supposed to be assertions but actually assert nothing, silently. In general, I prefer tools that don't assume I'm perfect. If I never made mistakes I wouldn't need to write tests in the first place, at least for code where I was the only maintainer.

What's more, this will also be a problem if new assertion methods are ever added to future versions of mock. Imagine developer A writes tests using a new assertion only available in mock v1.1; when she runs the tests, they work correctly. Developer B is working on the same code base, but forgot to upgrade and is using mock v1.0 that lacks the new assertion. When he runs the tests they are not actually testing what they seem to be testing. Oops.

The basic design flaw here is having a public API on objects that also create arbitrary attributes on demand. The whole public API of mock.Mock (assertions, attributes, etc.) should be exposed as module-level functions, so that typos or misremembering the API will result in a nice informative AttributeError. Until that happens, I will stick to mock.create_autospec, and avoid mock.Mock or mock.MagicMock.

1 comment:

  1. Had the same experience - and agree that separating mock/assertion methods from the actual mock objects would be better design. Also, you are out of luck if some of the mock-specific api "shadows" your actual API.