Testing with Python: DocTest
This post covers a fundamental portion of testing with Python: doctest. Other posts will cover other parts of testing with Python.
doctest is a Python module that will read all of the docstrings in your code
and execute anything in those docstrings that looks like Python code typed into
the interactive interpreter. Here’s a brief example:
If you were to save that as
doctest.py, you could tell Python to run doctest
against it, and Python would execute the statement
5 + 5. You told it that
the answer is
10, so it would expect the answer to be
10. It is, because
math. Here’s an example of a successful test.
Note that your
doctest.pyhas to be in a subfolder for this to work. You can’t run
python -m doctest -vat the same level as the Python file itself.
Cool, right? What about breaking it? Let’s change that 10 to an 11 and see the results. For reference, here’s what the test looks like now:
Our brains, of course, tell us that this is wrong. What does Python have to say about it?
This tells you exactly where and why a doctest failed. Cool, right?
Let’s see a slightly more involved example (but not much). You’ll need to clone the git repo that I’ll be using for all of the Python testing posts.
git clone https://github.com/supertylerc/tdd-examples
This repo has some basic structure built-in. Here’s what it looks like:
There are actually
.pycfiles and a
__pycache__folder (if you’re using Python 3) once you run any code, but I’ve omitted them for brevity.
As you can see, this is not a complex directory structure. Let’s look at the
tdd-examples/hello_world.py file. For reference, here it is:
This should be entirely self-documenting. There is an object and two methods.
You can even see how the object is instantiated from the
alone (they’re under the
From this, you can see that you can also test exceptions. You can use a
doctest directive to ignore the details of the exception (since
those will almost always vary) and instead match the fact that an exception
was raised and that it was of the correct type.
If you’re familiar with unit tests, you may be asking what good doctests are. The answer is pretty simple: documenting the usage of your code in docstrings gives you free tests. Consumers of your code will thank you because they can see examples of how to use your code…and know that those examples are valid and working (instead of missing a letter or space here or there). You will thank yourself because fewer issues will be raised saying that your documentation sucks and the examples don’t work.
If you’re not familiar with unit tests, I’ll be covering them in a future blog post.
doctest is not a good method for writing unit tests. They’re ugly to read
when they get complicated, and they won’t cover everything without being
really, really, really long. Another downside?
doctest treats the entire
test as a single test instead of as multiple tests. So if you look at the
docstring test for
DocTest.calculate, you’ll see that I’m testing for several
things–correct answers, operations, exceptions–but they’re all seen as a
single test. Although using the
python -m doctest -v syntax for running the
test will give you some insight as to where exactly the failure happens, other
testing frameworks won’t. And if you’re collecting metrics on failed tests,
it’s obviously going to make things look awful when only a single portion of
Do not use
doctest as a replacement for unit tests. Write unit tests! Use
doctest as a way to test your documentation and examples. Users will thank