Coverage.py is a tool for measuring code coverage of Python programs. Itmonitors your program, noting which parts of the code have been executed, thenanalyzes the source to identify code that could have been executed but was not.

Run tests that failed in the most recent test run. Similar to Python: Run Failed Tests on versions prior to 2021.9. Test: Rerun Last Run: Debug tests that were executed in the most recent test run. Test: Run All Tests: Run all discovered tests. Equivalent to Python: Run All Tests on versions prior to 2021.9. Test: Run Test at Cursor. 'Module implementing the command line entry point for executing tests. This module can be executed from the command line using the following approaches:: python -m robot.run python path/to/robot/run.py Instead of ``python`` it is possible to use also other Python interpreters. This module is also used by the installed ``robot`` start-up script. Run tests that failed in the most recent test run. Similar to Python: Run Failed Tests on versions prior to 2021.9. Test: Rerun Last Run: Debug tests that were executed in the most recent test run. Test: Run All Tests: Run all discovered tests. Equivalent to Python: Run All Tests on versions prior to 2021.9. Test: Run Test at Cursor.

Coverage measurement is typically used to gauge the effectiveness of tests. Itcan show which parts of your code are being exercised by tests, and which arenot.

The latest version is coverage.py 5.5, released February 28, 2021. It issupported on:

• Python versions 2.7, 3.5, 3.6, 3.7, 3.8, 3.9, and 3.10 alpha.

• PyPy2 7.3.3 and PyPy3 7.3.3.

For Enterprise¶

Available as part of the Tidelift Subscription.
Coverage and thousands of other packages are working withTidelift to deliver one enterprise subscription that covers all of the opensource you use. If you want the flexibility of open source and the confidenceof commercial-grade software, this is for you. Learn more.

Quick start¶

Getting started is easy:

1. Install coverage.py:

   For more details, see Installation.

2. Use coveragerun to run your test suite and gather data. However younormally run your test suite, you can run your test runner under coverage.If your test runner command starts with “python”, just replace the initial“python” with “coverage run”.

   Instructions for specific test runners:

   If you usually use:

   then you can run your tests under coverage with:

   Many people choose to use the pytest-cov plugin, but for mostpurposes, it is unnecessary.

   Change “python” to “coverage run”, so this:

   becomes:

   Nose has been unmaintained for a long time. You should seriouslyconsider adopting a different test runner.

   Change this:

   to:

   To limit coverage measurement to code in the current directory, and alsofind files that weren’t executed at all, add the --source=. argument toyour coverage command line.

3. Use coveragereport to report on the results:

4. For a nicer presentation, use coveragehtml to get annotated HTMLlistings detailing missed lines:

   Then open htmlcov/index.html in your browser, to see areport like this.

Using coverage.py¶

There are a few different ways to use coverage.py. The simplest is thecommand line, which lets you run your program and see the results.If you need more control over how your project is measured, you can use theAPI.

Some test runners provide coverage integration to make it easy to usecoverage.py while running tests. For example, pytest has the pytest-covplugin.

You can fine-tune coverage.py’s view of your code by directing it to ignoreparts that you know aren’t interesting. See Specifying source files and Excluding code from coverage.pyfor details.

Getting help¶

If the FAQ doesn’t answer your question, you can discusscoverage.py or get help using it on the Testing In Python mailing list.

Bug reports are gladly accepted at the GitHub issue tracker.GitHub also hosts the code repository.

Professional support for coverage.py is available as part of the TideliftSubscription.

I can be reached in a number of ways. I’m happy to answer questions aboutusing coverage.py.

More information¶

pytest allows to easily parametrize test functions.For basic docs, see Parametrizing fixtures and test functions.

In the following we provide some examples usingthe builtin mechanisms.

Generating parameters combinations, depending on command line¶

Let’s say we want to execute a test with different computationparameters and the parameter range shall be determined by a commandline argument. Let’s first write a simple (do-nothing) computation test:

Now we add a test configuration like this:

This means that we only run 2 tests if we do not pass --all:

We run only two computations, so we see two dots.let’s run the full monty:

As expected when running the full range of param1 valueswe’ll get an error on the last one.

Different options for test IDs¶

pytest will build a string that is the test ID for each set of values in aparametrized test. These IDs can be used with -k to select specific casesto run, and they will also identify the specific case when one is failing.Running pytest with --collect-only will show the generated IDs.

Numbers, strings, booleans and None will have their usual string representationused in the test ID. For other objects, pytest will make a string based onthe argument name:

In test_timedistance_v0, we let pytest generate the test IDs.

In test_timedistance_v1, we specified ids as a list of strings which wereused as the test IDs. These are succinct, but can be a pain to maintain.

In test_timedistance_v2, we specified ids as a function that can generate astring representation to make part of the test ID. So our datetime values use thelabel generated by idfn, but because we didn’t generate a label for timedeltaobjects, they are still using the default pytest representation:

In test_timedistance_v3, we used pytest.param to specify the test IDstogether with the actual data, instead of listing them separately.

A quick port of “testscenarios”¶

Here is a quick port to run tests configured with test scenarios,an add-on from Robert Collins for the standard unittest framework. Weonly have to work a bit to construct the correct arguments for pytest’sMetafunc.parametrize():

this is a fully self-contained example which you can run with:

If you just collect tests you’ll also nicely see ‘advanced’ and ‘basic’ as variants for the test function:

Note that we told metafunc.parametrize() that your scenario valuesshould be considered class-scoped. With pytest-2.3 this leads to aresource-based ordering.

Deferring the setup of parametrized resources¶

The parametrization of test functions happens at collectiontime. It is a good idea to setup expensive resources like DBconnections or subprocess only when the actual test is run.Here is a simple example how you can achieve that. This testrequires a db object fixture:

We can now add a test configuration that generates two invocations ofthe test_db_initialized function and also implements a factory thatcreates a database object for the actual test invocations:

Let’s first see how it looks like at collection time:

And then when we run the test:

The first invocation with db'DB1' passed while the second with db'DB2' failed. Our db fixture function has instantiated each of the DB values during the setup phase while the pytest_generate_tests generated two according calls to the test_db_initialized during the collection phase.

Indirect parametrization¶

Using the indirect=True parameter when parametrizing a test allows toparametrize a test with a fixture receiving the values before passing them to atest:

This can be used, for example, to do more expensive setup at test run time inthe fixture, rather than having to run those setup steps at collection time.

Apply indirect on particular arguments¶

Very often parametrization uses more than one argument name. There is opportunity to apply indirectparameter on particular arguments. It can be done by passing list or tuple ofarguments’ names to indirect. In the example below there is a function test_indirect which usestwo fixtures: x and y. Here we give to indirect the list, which contains the name of thefixture x. The indirect parameter will be applied to this argument only, and the value awill be passed to respective fixture function:

The result of this test will be successful:

Parametrizing test methods through per-class configuration¶

Here is an example pytest_generate_tests function implementing aparametrization scheme similar to Michael Foord’s unittestparametrizer but in a lot less code:

Our test generator looks up a class-level definition which specifies whichargument sets to use for each test function. Let’s run it:

Indirect parametrization with multiple fixtures¶

Usage

Here is a stripped down real-life example of using parametrizedtesting for testing serialization of objects between different pythoninterpreters. We define a test_basic_objects function whichis to be run with different sets of arguments for its three arguments:

• python1: first python interpreter, run to pickle-dump an object to a file

• python2: second interpreter, run to pickle-load an object from a file

• obj: object to be dumped/loaded

Running it results in some skips if we don’t have all the python interpreters installed and otherwise runs all combinations (3 interpreters times 3 interpreters times 3 objects to serialize/deserialize):

Indirect parametrization of optional implementations/imports¶

If you want to compare the outcomes of several implementations of a givenAPI, you can write test functions that receive the already imported implementationsand get skipped in case the implementation is not importable/available. Let’ssay we have a “base” implementation and the other (possibly optimized ones)need to provide similar results:

And then a base implementation of a simple function:

And an optimized version:

And finally a little test module:

If you run this with reporting for skips enabled:

You’ll see that we don’t have an opt2 module and thus the second test runof our test_func1 was skipped. A few notes:

• the fixture functions in the conftest.py file are “session-scoped” because wedon’t need to import more than once

• if you have multiple test functions and a skipped import, you will seethe [1] count increasing in the report

• you can put @pytest.mark.parametrize styleparametrization on the test functions to parametrize input/outputvalues as well.

Set marks or test ID for individual parametrized test¶

Use pytest.param to apply marks or set test ID to individual parametrized test.For example:

Unittest Htmltestrunner

In this example, we have 4 parametrized tests. Except for the first test,we mark the rest three parametrized tests with the custom marker basic,and for the fourth test we also use the built-in mark xfail to indicate thistest is expected to fail. For explicitness, we set test ids for some tests.

Then run pytest with verbose mode and with only the basic marker:

As the result:

• Four tests were collected

• One test was deselected because it doesn’t have the basic mark.

• Three tests with the basic mark was selected.

• The test test_eval[1+7-8] passed, but the name is autogenerated and confusing.

• The test test_eval[basic_2+4] passed.

• The test test_eval[basic_6*9] was expected to fail and did fail.

Parametrizing conditional raising¶

Use pytest.raises() with thepytest.mark.parametrize decorator to write parametrized testsin which some tests raise exceptions and others do not.

It is helpful to define a no-op context manager does_not_raise to serveas a complement to raises. For example:

In the example above, the first three test cases should run unexceptionally,while the fourth should raise ZeroDivisionError.

If you’re only supporting Python 3.7+, you can simply use nullcontextto define does_not_raise:

Or, if you’re supporting Python 3.3+ you can use:

Html Test Runner Python Tutorial

Or, if desired, you can pipinstallcontextlib2 and use: