Here's a rather silly example of two unit tests using mock.
One would expectimport unittest import mock class C: def function(self, x): pass class Tests(unittest.TestCase): def test_positive(self): C2 = mock.Mock(spec=C) obj = C2() obj.function(1) obj.function.assert_called_once_with(1) def test_negative(self): C2 = mock.Mock(spec=C) obj = C2() obj.function(1) obj.function.assert_not_called() if __name__ == '__main__': unittest.main()
test_negativeto fail, but in fact both tests pass:
Oops.$ python mocktests.py .. ------------------------------------ Ran 2 tests in 0.001s OK
If you've used mock before, you probably know what's going on. The mock library creates new attributes on demand if they don't exist. Thus:
Once again, oops. I had invented a new assertion method>>> import mock >>> obj = mock.Mock() >>> obj.assert_called_once_with <bound method Mock.assert_called_once_with of <mock.Mock object at 0x7f6e1c18a990>> >>> obj.assert_not_called <Mock name='mock.assert_not_called' id='140110894509008'>
assert_not_calledwhich doesn't actually exist, and mock had happily created a new object for me. My test was completely broken. My mistake, and therefore my fault. And in fact the mock documentation does mention the possibility of this happening, deep within the bowels of the API documentation: "Because mocks auto-create attributes on demand, and allow you to call them with arbitrary arguments, if you misspell one of these assert methods then your assertion is gone." An appropriate fix is provided, the
mock.create_autospecAPI. It would have been better to include this suggestion in the intro documentation; better yet would be preventing the problem in the first place.
By their very nature, test assertions do nothing silently in the expected case. It's thus quite dangerous to have a library specifically intended for testing where typos create calls that are supposed to be assertions but actually assert nothing, silently. In general, I prefer tools that don't assume I'm perfect. If I never made mistakes I wouldn't need to write tests in the first place, at least for code where I was the only maintainer.
What's more, this will also be a problem if new assertion methods are ever added to future versions of mock. Imagine developer A writes tests using a new assertion only available in mock v1.1; when she runs the tests, they work correctly. Developer B is working on the same code base, but forgot to upgrade and is using mock v1.0 that lacks the new assertion. When he runs the tests they are not actually testing what they seem to be testing. Oops.
The basic design flaw here is having a public API on objects that also create arbitrary attributes on demand. The whole public API of
mock.Mock(assertions, attributes, etc.) should be exposed as module-level functions, so that typos or misremembering the API will result in a nice informative
AttributeError. Until that happens, I will stick to
mock.create_autospec, and avoid