Python PyTest

Reference

Getting Started With Testing in Python
Test Your Python Apps

Effective Python Testing With Pytest
Test-Driven Development With PyTest
TDD Project Sample Code


PyTest plugins

How to install and use plugins

pytest-randomly

pytest-randomly does something seemingly simple but with valuable effect: It forces your tests to run in a random order. pytest always collects all the tests it can find before running them, so pytest-randomly shuffles that list of tests just before execution.

This is a great way to uncover tests that depend on running in a specific order, which means they have a stateful dependency on some other test. If you built your test suite from scratch in pytest, then this isn’t very likely. It’s more likely to happen in test suites that you migrate to pytest.

The plugin will print a seed value in the configuration description. You can use that value to run the tests in the same order as you try to fix the issue.

pytest-cov

If you measure how well your tests cover your implementation code, you likely use the coverage package. pytest-cov integrates coverage, so you can run pytest --cov to see the test coverage report.

pytest-django

pytest-django provides a handful of useful fixtures and marks for dealing with Django tests. You saw the django_db mark earlier in this tutorial, and the rf fixture provides direct access to an instance of Django’s RequestFactory. The settings fixture provides a quick way to set or override Django settings. This is a great boost to your Django testing productivity!

If you’re interested in learning more about using pytest with Django, then check out How to Provide Test Fixtures for Django Models in Pytest.

pytest-bdd

pytest can be used to run tests that fall outside the traditional scope of unit testing. Behavior-driven development (BDD) encourages writing plain-language descriptions of likely user actions and expectations, which you can then use to determine whether to implement a given feature. pytest-bdd helps you use Gherkin to write feature tests for your code.

You can see which other plugins are available for pytest with this extensive list of third-party plugins.


Testing Your Code

There are many ways to test your code. In this tutorial, you’ll learn the techniques from the most basic steps and work towards advanced methods.


Automated vs. Manual Testing

Exploratory testing is a form of testing that is done without a plan. In an exploratory test, you’re just exploring the application.

To have a complete set of manual tests, all you need to do is make a list of all the features your application has, the different types of input it can accept, and the expected results. Now, every time you make a change to your code, you need to go through every single item on that list and check it.


Unit Tests vs. Integration Tests

Think of how you might test the lights on a car. You would turn on the lights (known as the test step) and go outside the car or ask a friend to check that the lights are on (known as the test assertion). Testing multiple components is known as integration testing.

A unit test is a smaller test, one that checks that a single component operates in the right way. A unit test helps you to isolate what is broken in your application and fix it faster.

You have just seen two types of tests:

  1. An integration test checks that components in your application operate with each other.
  2. A unit test checks a small component in your application.

Choosing a Test Runner

There are many test runners available for Python. The one built into the Python standard library is called unittest. In this tutorial, you will be using unittest test cases and the unittest test runner. The principles of unittest are easily portable to other frameworks. The three most popular test runners are:

  • unittest
  • nose or nose2
  • pytest

How to Structure a Simple Test

Before you dive into writing tests, you’ll want to first make a couple of decisions:

  1. What do you want to test?
  2. Are you writing a unit test or an integration test?

Then the structure of a test should loosely follow this workflow:

  1. Create your inputs
  2. Execute the code being tested, capturing the output
  3. Compare the output with an expected result

PyTest Introduction

pytest presents the test results differently than unittest. The report shows:

  1. The system state, including which versions of Python, pytest, and any plugins you have installed
  2. The rootdir, or the directory to search under for configuration and tests
  3. The number of tests the runner discovered

The output then indicates the status of each test using a syntax similar to unittest:

  • A dot (.) means that the test passed.
  • An F means that the test has failed.
  • An E means that the test raised an unexpected exception.

For tests that fail, the report gives a detailed breakdown of the failure. In the example above, the test failed because assert False always fails. Finally, the report gives an overall status report of the test suite.


PyTest Summary

  • Fixtures for handling test dependencies, state, and reusable functionality
  • Marks for categorizing tests and limiting access to external resources
  • Parametrization for reducing duplicated code between tests
  • Durations to identify your slowest tests
  • Plugins for integrating with other frameworks and testing tools

Fixtures: Managing State and Dependencies

pytest fixtures are a way of providing data, test doubles, or state setup to your tests. Fixtures are functions that can return a wide range of values. Each test that depends on a fixture must explicitly accept that fixture as an argument.


When to Create Fixtures

The directory structure should be the following:

1
2
3
4
5
6
.
├── display.py
├── excel.py
└── tests_ver1
├── test_display.py
└── test_excel.py

Imagine you’re writing a function, format_data_for_display(), to process the data returned by an API endpoint. The data represents a list of people, each with a given name, family name, and job title. The function should output a list of strings that include each person’s full name (their given_name followed by their family_name), a colon, and their title. To test this, you might write the following code:

display.py

1
2
3
4
def format_data_for_display(people):
"""Output a list of strings that include each person’s full name (their given_name followed by their family_name), a colon, and their title.
"""
return [f"{i['given_name']} {i['family_name']}: {i['title']}" for i in people]

test_ver1/test_display.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from display import format_data_for_display

import pytest

def test_format_data_for_display():
people = [
{
"given_name": "Alfonsa",
"family_name": "Ruiz",
"title": "Senior Software Engineer",
},
{
"given_name": "Sayid",
"family_name": "Khan",
"title": "Project Manager",
},
]

assert format_data_for_display(people) == [
"Alfonsa Ruiz: Senior Software Engineer",
"Sayid Khan: Project Manager",
]

Now suppose you need to write another function to transform the data into comma-separated values for use in Excel. The test would look awfully similar:

excel.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from openpyxl import Workbook

def format_data_for_excel(people):
"""Output an Excel that include each person’s full name (their given_name followed by their family_name), a colon, and their title.
"""
workbook = Workbook()
sheet = workbook.active
sheet['A1'] = 'given'
sheet['B1'] = 'family'
sheet['C1'] = 'title'
for i in people:
sheet.append(list(i.values()))
# workbook.save(filename="people.xlsx")
return [value for value in sheet.iter_rows(values_only=True)]

tests_ver1/test_excel.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from excel import format_data_for_excel

import pytest

def test_format_data_for_excel():
people = [
{
"given_name": "Alfonsa",
"family_name": "Ruiz",
"title": "Senior Software Engineer",
},
{
"given_name": "Sayid",
"family_name": "Khan",
"title": "Project Manager",
},
]

assert format_data_for_excel(people) == [
('given', 'family', 'title'),
('Alfonsa', 'Ruiz', 'Senior Software Engineer'),
('Sayid', 'Khan', 'Project Manager')
]

Run the following command:

1
python3 -m pytest tests_ver1 -v

Output:

1
2
3
4
5
6
7
8
9
10
11
================================================================= test session starts =================================================================
platform darwin -- Python 3.9.4, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/zacks/Desktop/Code/Python/Exercise/PyTest/format_data
plugins: cov-2.12.1, instafail-0.4.2, anyio-2.2.0, xdist-2.3.0, timeout-1.4.2, forked-1.3.0, django-4.4.0, flakes-4.0.3, pep8-1.0.6
collected 2 items

tests_ver1/test_display.py::test_format_data_for_display PASSED [ 50%]
tests_ver1/test_excel.py::test_format_data_for_excel PASSED [100%]

================================================================== 2 passed in 0.34s ==================================================================

If you find yourself writing several tests that all make use of the same underlying test data, then a fixture may be in your future.

Let’s make a new test file under the directory tests_ver2

1
2
3
4
5
6
7
8
.
├── display.py
├── excel.py
├── tests_ver1
│ ├── test_display.py
│ └── test_excel.py
└── tests_ver2
└── test_all.py

You can pull the repeated data into a single function decorated with @pytest.fixture to indicate that the function is a pytest fixture:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from display import format_data_for_display
from excel import format_data_for_excel

import pytest

@pytest.fixture
def example_people_data():
return [
{
"given_name": "Alfonsa",
"family_name": "Ruiz",
"title": "Senior Software Engineer",
},
{
"given_name": "Sayid",
"family_name": "Khan",
"title": "Project Manager",
},
]

You can use the fixture by adding it as an argument to your tests. Its value will be the return value of the fixture function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Unit test for format_data_for_display
def test_format_data_for_display(example_people_data):
assert format_data_for_display(example_people_data) == [
"Alfonsa Ruiz: Senior Software Engineer",
"Sayid Khan: Project Manager",
]

# Unit test for format_data_for_excel
def test_format_data_for_excel(example_people_data):
assert format_data_for_excel(example_people_data) == [
('given', 'family', 'title'),
('Alfonsa', 'Ruiz', 'Senior Software Engineer'),
('Sayid', 'Khan', 'Project Manager')
]

Run the following command:

1
python3 -m pytest tests_ver2 -v

Output:

1
2
3
4
5
6
7
8
9
10
11
================================================================= test session starts =================================================================
platform darwin -- Python 3.9.4, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/local/opt/python@3.9/bin/python3.9
cachedir: .pytest_cache
rootdir: /Users/zacks/Desktop/Code/Python/Exercise/PyTest/format_data
plugins: cov-2.12.1, instafail-0.4.2, anyio-2.2.0, xdist-2.3.0, timeout-1.4.2, forked-1.3.0, django-4.4.0, flakes-4.0.3, pep8-1.0.6
collected 2 items

tests_ver2/test_all.py::test_format_data_for_display PASSED [ 50%]
tests_ver2/test_all.py::test_format_data_for_excel PASSED [100%]

================================================================== 2 passed in 0.33s ==================================================================

Each test is now notably shorter but still has a clear path back to the data it depends on. Be sure to name your fixture something specific. That way, you can quickly determine if you want to use it when writing new tests in the future!


When to Avoid Fixtures

Fixtures are great for extracting data or objects that you use across multiple tests. They aren’t always as good for tests that require slight variations in the data. Littering your test suite with fixtures is no better than littering it with plain data or objects. It might even be worse because of the added layer of indirection.

As with most abstractions, it takes some practice and thought to find the right level of fixture use.


Fixtures at Scale

As you extract more fixtures from your tests, you might see that some fixtures could benefit from further extraction. Fixtures are modular, so they can depend on other fixtures. You may find that fixtures in two separate test modules share a common dependency. What can you do in this case?

You can move fixtures from test modules into more general fixture-related modules. That way, you can import them back into any test modules that need them. This is a good approach when you find yourself using a fixture repeatedly throughout your project.

pytest looks for conftest.py modules throughout the directory structure. Each conftest.py provides configuration for the file tree pytest finds it in. You can use any fixtures that are defined in a particular conftest.py throughout the file’s parent directory and in any subdirectories. This is a great place to put your most widely used fixtures.

Another interesting use case for fixtures is in guarding access to resources. Imagine that you’ve written a test suite for code that deals with API calls. You want to ensure that the test suite doesn’t make any real network calls, even if a test accidentally executes the real network call code. pytest provides a monkeypatch fixture to replace values and behaviors, which you can use to great effect:

1
2
3
4
5
6
7
8
9
10
# conftest.py

import pytest
import requests

@pytest.fixture(autouse=True)
def disable_network_calls(monkeypatch):
def stunted_get():
raise RuntimeError("Network access not allowed during testing!")
monkeypatch.setattr(requests, "get", lambda *args, **kwargs: stunted_get())

By placing disable_network_calls() in conftest.py and adding the autouse=True option, you ensure that network calls will be disabled in every test across the suite. Any test that executes code calling requests.get() will raise a RuntimeError indicating that an unexpected network call would have occurred.


Parametrization: Combining Tests

You saw earlier in this tutorial how pytest fixtures can be used to reduce code duplication by extracting common dependencies. Fixtures aren’t quite as useful when you have several tests with slightly different inputs and expected outputs. In these cases, you can parametrize a single test definition, and pytest will create variants of the test for you with the parameters you specify.

Imagine you’ve written a function to tell if a string is a palindrome. An initial set of tests could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def test_is_palindrome_empty_string():
assert is_palindrome("")

def test_is_palindrome_single_character():
assert is_palindrome("a")

def test_is_palindrome_mixed_casing():
assert is_palindrome("Bob")

def test_is_palindrome_with_spaces():
assert is_palindrome("Never odd or even")

def test_is_palindrome_with_punctuation():
assert is_palindrome("Do geese see God?")

def test_is_palindrome_not_palindrome():
assert not is_palindrome("abc")

def test_is_palindrome_not_quite():
assert not is_palindrome("abab")

All of these tests except the last two have the same shape:

1
2
def test_is_palindrome_<in some situation>():
assert is_palindrome("<some string>")

You can use @pytest.mark.parametrize() to fill in this shape with different values, reducing your test code significantly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@pytest.mark.parametrize("palindrome", [
"",
"a",
"Bob",
"Never odd or even",
"Do geese see God?",
])
def test_is_palindrome(palindrome):
assert is_palindrome(palindrome)

@pytest.mark.parametrize("non_palindrome", [
"abc",
"abab",
])
def test_is_palindrome_not_palindrome(non_palindrome):
assert not is_palindrome(non_palindrome)

The first argument to parametrize() is a comma-delimited string of parameter names. The second argument is a list of either tuples or single values that represent the parameter value(s). You could take your parametrization a step further to combine all your tests into one:

1
2
3
4
5
6
7
8
9
10
11
@pytest.mark.parametrize("maybe_palindrome, expected_result", [
("", True),
("a", True),
("Bob", True),
("Never odd or even", True),
("Do geese see God?", True),
("abc", False),
("abab", False),
])
def test_is_palindrome(maybe_palindrome, expected_result):
assert is_palindrome(maybe_palindrome) == expected_result

Even though this shortened your code, it’s important to note that in this case, it didn’t do much to clarify your test code. Use parametrization to separate the test data from the test behavior so that it’s clear what the test is testing!


Marks: Categorizing Tests

In any large test suite, some of the tests will inevitably be slow. They might test timeout behavior, for example, or they might exercise a broad area of the code. Whatever the reason, it would be nice to avoid running all the slow tests when you’re trying to iterate quickly on a new feature.

pytest enables you to define categories for your tests and provides options for including or excluding categories when you run your suite. You can mark a test with any number of categories.

Marking tests is useful for categorizing tests by subsystem or dependencies. If some of your tests require access to a database, for example, then you could create a @pytest.mark.database_access mark for them.

Pro tip: Because you can give your marks any name you want, it can be easy to mistype or misremember the name of a mark. pytest will warn you about marks that it doesn’t recognize.

The --strict-markers flag to the pytest command ensures that all marks in your tests are registered in your pytest configuration. It will prevent you from running your tests until you register any unknown marks.

For more information on registering marks, check out the pytest documentation.

When the time comes to run your tests, you can still run them all by default with the pytest command. If you’d like to run only those tests that require database access, then you can use pytest -m database_access. To run all tests except those that require database access, you can use pytest -m "not database_access". You can even use an autouse fixture to limit database access to those tests marked with database_access.

Some plugins expand on the functionality of marks by guarding access to resources. The pytest-django plugin provides a django_db mark. Any tests without this mark that try to access the database will fail. The first test that tries to access the database will trigger the creation of Django’s test database.

The requirement that you add the django_db mark nudges you toward stating your dependencies explicitly. That’s the pytest philosophy, after all! It also means that you can run tests that don’t rely on the database much more quickly, because pytest -m "not django_db" will prevent the test from triggering database creation. The time savings really add up, especially if you’re diligent about running your tests frequently.

pytest provides a few marks out of the box:

  • skip skips a test unconditionally.
  • skipif skips a test if the expression passed to it evaluates to True.
  • xfail indicates that a test is expected to fail, so if the test does fail, the overall suite can still result in a passing status.
  • parametrize (note the spelling) creates multiple variants of a test with different values as arguments. You’ll learn more about this mark shortly.

You can see a list of all the marks pytest knows about by running pytest --markers.


Durations Reports: Fighting Slow Tests

Each time you switch contexts from implementation code to test code, you incur some overhead. If your tests are slow to begin with, then overhead can cause friction and frustration.

You read earlier about using marks to filter out slow tests when you run your suite. If you want to improve the speed of your tests, then it’s useful to know which tests might offer the biggest improvements. pytest can automatically record test durations for you and report the top offenders.

Use the --durations option to the pytest command to include a duration report in your test results. --durations expects an integer value n and will report the slowest n number of tests. The output will follow your test results:

1
2
3
4
5
$ pytest --durations=3
3.03s call test_code.py::test_request_read_timeout
1.07s call test_code.py::test_request_connection_timeout
0.57s call test_code.py::test_database_read
======================== 7 passed in 10.06s ==============================

Each test that shows up in the durations report is a good candidate to speed up because it takes an above-average amount of the total testing time.

Be aware that some tests may have an invisible setup overhead. You read earlier about how the first test marked with django_db will trigger the creation of the Django test database. The durations report reflects the time it takes to set up the database in the test that triggered the database creation, which can be misleading.