This extension allows you to run your Python Unittest, Pytest or Testplantests with the Test Explorer UI.

• Python Html Test Runner
• Intellij Python Test Runner
• Web Python Runner
• Python Compilers Idle

Your code isn’t a unit test so you wouldn’t be able to run it as a unit test. To use your code run it using a ‘Pure Python’ run configuration, the easiest way to do this is to right-click anywhere in your code, and to click ‘Run ‘. If you’d like to know more about unit testing you could read this tutorial: https://jeffknupp. Project: pywren-ibm-cloud Author: pywren File: tests.py License: Apache License 2.0.

Getting started

• Install the extension
• Configure Visual Studio Code to discover your tests(see Configuration section and documentation of a test framework of your choice:
• Open Test View sidebar
• Run your tests using the icon in the Test Explorer

Features

• Shows a Test Explorer in the Test view in VS Code's sidebar with all detected tests and suites and their state
• Convenient error reporting during test discovery
• Unittest, Pytest and Testplan debugging
• Shows a failed test's log when the test is selected in the explorer
• Re-run tests on save
• Supports multi-root workspaces
• Supports Unittest, Pytest and Testplan test frameworks and their plugins

Comparison to Python extension's Test View

• Better error reporting during the discovery stage. In case of errors, you will see such tests in an errored state, and by clicking on them, a complete error message would be shown in the Output panel. Python Extension, at best, won't show your tests that contain errors (syntax errors and invalid imports, for example).
• Works better with pytest plugins - Tavern, for example. Python Extension won't discover these tests.
• Based on Test Explorer UI. This fact may be useful if you have a workspace with projects in different languages/frameworks. Test Explorer UI has a lot of plugins, and you can conveniently discover and run tests at the same View.
• Shows you errors and a complete output of your tests just by clicking on a failed test.
• Shows only relevant folders from your workspace. Showing all workspace folders, as the Python Extension is doing, can be an issue when you have multiple workspace folders, but only a couple of them have any tests.
• There might be some configurations when this extension will discover and run your tests, but Python Extension - won't.
• User experience with both extensions is highly subjective. However, you might like the user interface of this extension better. Also, each discovery, test execution, and test cancellation won't require you to select a folder when you have multiple in your workspace.

Configuration

By default the extension uses the configuration from Python extension for Visual Studio Code.To configure Python for your project see Getting Started with Python in VS Code.

However, test framework used by this extension can be overridden by pythonTestExplorer.testFramework configuration property.Right now, the two available option are unittest, pytest and testplan. When property is set to null, the configuration from Python extension is used.

Configuring Python test discovery and execution

List of currently used properties:

Configuration supports placeholders for workspace folder as ${workspaceFolder} and environment variables in a form of ${env:YOUR_ENVIRONMENT_VARIABLE}.

Configuring Test Explorer UI

The following configuration properties are provided by Test Explorer UI:

See Test Explorer UI documentation for the latest changes in configuration.

Configuring debug

The extension will look for a configuration in launch.json with 'type': 'python' and 'request': 'test' to load any of the following options during debugging

• name
• console
• env
• stopOnEntry
• showReturnValue
• redirectOutput
• debugStdLib
• justMyCode
• subProcess
• envFile

For example,

FAQ

1. Disable duplicated Code Lenses

Test Explorer UI provides a set of Code Lenses to run and debug tests. However, the Python extension also provides a set of its own. As of now, there are no way to disable Code Lenses from the Python extension, see https://github.com/microsoft/vscode-python/issues/10898. If you use only Python Test Explorer to run your tests, you can disable testing functionality by the Python extension with the following settings

2. How to use this extesion to run Django tests?

You can use pytest to discover and run Django tests with this extension. For this, install pytest-django package and follow its Getting started guide.

3. How to re-run tests on save?

Test Explorer UI allows to run tests on saving files in a workspace. To enable autorun, right-click on a test suite and select 'Enable autorun'. After that, any changes to files in a workspace will re-execute tests when files are saved. See the example below.

You can enable autorun for all tests by clicking on three dots in the Test explorer UI bar.

Python Html Test Runner

Troubleshooting

Whether no tests were discovered in the Test Explorer view or anything else doesn't work as expected, you can see logging output selecting Python Test Adapter Log in the Output section.

Questions, issues, feature requests, and contributions

• If you're happy using this extension - star GitHub repo and share your positive feedback in VSCode Marketplace.
• If you have any question or a problem with the extension, please file an issue. Make sure, to include information on Python version, test framework you using, what plugins are installed (mostly for pytest), and what do you see in logs (Python Test Adapter Log in the Output section).
• Contributions are always welcome! Please see contributing guide for more details.

This articles serves as a guide to testing Flask applications with pytest.

We'll first look at why testing is important for creating maintainable software and what you should focus on when testing. Then, we'll detail how to:

• Create and run unit and functional tests with pytest
• Utilize fixtures to initialize the state for test functions
• Check the coverage of the tests using coverage.py

The source code (along with detailed installation instructions) for the Flask app being tested in this article can be found on GitLab at https://gitlab.com/patkennedy79/flask_user_management_example.

Contents

Objectives

By the end of this article, you should be able to:

1. Explain what to test in a Flask app
2. Describe the differences between pytest and unittest
3. Write unit and functional test functions with pytest
4. Run tests with pytest
5. Create fixtures for initializing the state for test functions
6. Determine code coverage of your tests with coverage.py

Why Write Tests?

In general, testing helps ensure that your app will work as expected for your end users.

Software projects with high test coverage are never perfect, but it's a good initial indicator of the quality of the software. Additionally, testable code is generally a sign of a good software architecture, which is why advanced developers take testing into account throughout the entire development lifecycle.

Tests can be considered at three levels:

• Unit
• Functional (or integration)
• End-to-end

Unit tests test the functionality of an individual unit of code isolated from its dependencies. They are the first line of defense against errors and inconsistencies in your codebase. They test from the inside out, from the programmer's point of view.

Functional tests test multiple components of a software product to make sure the components are working together properly. Typically, these tests are focused on functionality that the user will be utilizing. They test from the outside in, from the end user's point of view.

Both unit and Functional testing are fundamental parts of the Test-Driven Development (TDD) process.

Testing improves the maintainability of your code.

Maintainability refers to making bug fixes or enhancements to your code or to another developer needing to update your code at some point in the future.

Testing should be combined with a Continuous Integration (CI) process to ensure that your tests are constantly executing, ideally on each commit to your repository. A solid suite of tests can be critical to catching defects quickly and early in the development process before your end users come across them in production.

What to Test?

What should you test?

Again, unit tests should focus on testing small units of code in isolation.

For example, in a Flask app, you may use unit tests to test:

1. Database models
2. Utility functions that your view functions call

Functional tests, meanwhile, should focus on how the view functions operate.

For example:

1. Nominal conditions (GET, POST, etc.) for a view function
2. Invalid HTTP methods are handled properly for a view function
3. Invalid data is passed to a view function

Focus on testing scenarios that the end user will interact with. The experience that the users of your product have is paramount!

pytest vs. unittest

pytest is a test framework for Python used to write, organize, and run test cases. After setting up your basic test structure, pytest makes it really easy to write tests and provides a lot of flexibility for running the tests. pytest satisfies the key aspects of a good test environment:

• tests are fun to write
• tests can be written quickly by using helper functions (fixtures)
• tests can be executed with a single command
• tests run quickly

pytest is incredible! I highly recommend using it for testing any application or script written in Python.

If you're interested in really learning all the different aspects of pytest, I highly recommend the Python Testing with pytest book by Brian Okken.

Python has a built-in test framework called unittest, which is a great choice for testing as well. The unittest module is inspired by the xUnit test framework.

It provides the following:

• tools for building unit tests, including a full suite of assert statements for performing checks
• structure for developing unit tests and unit test suites
• test runner for executing tests

The main differences between pytest and unittest:

Either framework is good for testing a Flask project. However, I prefer pytest since it:

1. Requires less boilerplate code so your test suites will be more readable.
2. Supports the plain assert statement, which is far more readable and easier to remember compared to the assertSomething methods -- like assertEquals, assertTrue, and assertContains -- in unittest.
3. Is updated more frequently since it's not part of the Python standard library.
4. Simplifies setting up and tearing down test state.
5. Uses a functional approach.
6. Supports fixtures.

Testing

Project Structure

I like to organize all the test cases in a separate 'tests' folder at the same level as the application files.

Additionally, I really like differentiating between unit and functional tests by splitting them out as separate sub-folders. This structure gives you the flexibility to easily run just the unit tests (or just the functional tests, for that matter).

Here's an example of the structure of the 'tests' directory:

And, here's how the 'tests' folder fits into a typical Flask project with blueprints:

Unit Test Example

The first test that we're going to write is a unit test for project/models.py, which contains the SQLAlchemy interface to the database.

This test doesn't access the underlying database; it only checks the interface class used by SQLAlchemy.

Since this test is a unit test, it should be implemented in tests/unit/test_models.py:

Let's take a closer look at this test.

After the import, we start with a description of what the test does:

Why include so many comments for a test function?

I've found that tests are one of the most difficult aspects of a project to maintain. Often, the code (including the level of comments) for test suites is nowhere near the level of quality as the code being tested.

A common structure used to describe what each test function does helps with maintainability by making it easier for a someone (another developer, your future self) to quickly understand the purpose of each test.

A common practice is to use the GIVEN-WHEN-THEN structure:

• GIVEN - what are the initial conditions for the test?
• WHEN - what is occurring that needs to be tested?
• THEN - what is the expected response?

For more, review the GivenWhenThen article by Martin Fowler and the Python Testing with pytest book by Brian Okken.

Next, we have the actual test:

After creating a new user with valid arguments to the constructor, the properties of the user are checked to make sure it was created properly.

Functional Test Examples

The second test that we're going to write is an functional test for project/recipes/routes.py, which contains the view functions for the recipes blueprint.

Since this test is a functional test, it should be implemented in tests/functional/test_recipes.py:

This project uses the Application Factory Pattern to create the Flask application. Therefore, the create_app() function needs to first be imported:

The test function, test_home_page(), starts with the GIVEN-WHEN-THEN description of what the test does. Next, a Flask application (flask_app) is created:

In order to create the proper environment for testing, Flask provides a test_client helper. This creates a test version of our Flask application, which we used to make a GET call to the '/' URL. We then check that the status code returned is OK (200) and that the response contained the following strings:

• Welcome to the Flask User Management Example!
• Need an account?
• Existing user?

These checks match with what we expect the user to see when we navigate to the '/' URL:

Intellij Python Test Runner

An example of an off-nominal functional test would be to utilize an invalid HTTP method (POST) when accessing the '/' URL:

This test checks that a POST request to the '/' URL results in an error code of 405 (Method Not Allowed) being returned.

Take a second to review the two functional tests.. do you see some duplicate code between these two test functions? Do you see a lot of code for initializing the state needed by the test functions? We can use fixtures to address these issues.

Fixtures

Fixtures initialize tests to a known state in order to run tests in a predictable and repeatable manner.

xUnit

The classic approach to writing and executing tests follows the the xUnit type of test framework, where each test runs as follows:

1. SetUp()
2. ..run the test case..
3. TearDown()

The SetUp() and TearDown() methods always run for each unit test within a test suite. This approach results in the same initial state for each test within a test suite, which doesn't provide much flexibility.

Advantages of Fixtures

The test fixture approach provides much greater flexibility than the classic Setup/Teadown approach.

pytest-flask facilitates testing Flask apps by providing a set of common fixtures used for testing Flask apps. This library is not used in this tutorial, as I want to show how to create the fixtures that help support testing Flask apps.

First, fixtures are defined as functions (that should have a descriptive names for their purpose).

Second, multiple fixtures can be run to set the initial state for a test function. In fact, fixtures can even call other fixtures! So, you can compose them together to create the required state.

Finally, fixtures can be run with different scopes:

• function - run once per test function (default scope)
• class - run once per test class
• module - run once per module (e.g., a test file)
• session - run once per session

For example, if you have a fixture with module scope, that fixture will run once (and only once) before the test functions in the module run.

Fixtures should be created in tests/conftest.py.

Unit Test Example

To help facilitate testing the User class in project/models.py, we can add a fixture to tests/conftest.py that is used to create a User object to test:

The @pytest.fixture decorator specifies that this function is a fixture with module-level scope. In other words, this fixture will be called one per test module.

Scp containment breach god mode. This fixture, new_user, creates an instance of User using valid arguments to the constructor. user is then passed to the test function (return user).

We can simplify the test_new_user() test function from earlier by using the new_user fixture in tests/unit/test_models.py:

By using a fixture, the test function is reduced to the assert statements that perform the checks against the User object.

Functional Test Examples

Fixture

To help facilitate testing all the view functions in the Flask project, a fixture can be created in tests/conftest.py:

This fixture creates the test client using a context manager:

Next, the application context is pushed onto the stack for use by the test functions:

To learn more about the Application context in Flask, refer to the following blog posts:

• Basics: Understanding the Application and Request Contexts in Flask
• Advanced: Deep Dive into Flask's Application and Request Contexts

The yield testing_client statement means that execution is being passed to the test functions.

Using the Fixture

We can simplify the functional tests from earlier with the test_client fixture in tests/functional/test_recipes.py:

Notice how much duplicate code is eliminated by using the test_client fixture? By utilizing the test_client fixture, each test function is simplified down the HTTP call (GET or POST) and the assert that checks the response.

I really find that using fixtures helps to focus the test function on actually doing the testing, as the test initialization is handled in the fixture.

Running the Tests

To run the tests, run pytest in the top-level folder for the Flask project:

To see more details on the tests that were run:

If you only want to run a specific type of test:

• pytest tests/unit/
• pytest tests/functional/

Fixtures in Action

To really get a sense of when the test_client() fixture is run, pytest can provide a call structure of the fixtures and tests with the '--setup-show' argument:

The test_client fixture has a 'module' scope, so it's executed prior to the two _with_fixture tests in tests/functional/test_recipes.py.

If you change the scope of the test_client fixture to a 'function' scope:

Then the test_client fixture will run prior to each of the two _with_fixture tests:

Since we want the test_client fixture to only be run once in this module, revert the scope back to 'module'.

Code Coverage

When developing tests, it's nice to get an understanding of how much of the source code is actually tested. This concept is known as code coverage.

I need to be very clear that having a set of tests that covers 100% of the source code is by no means an indicator that the code is properly tested.

This metric means that there are a lot of tests and a lot of effort has been put into developing the tests. The quality of the tests still needs to be checked by code inspection.

The other extreme where this is a minimal set (or none!) of tests is a very bad indicator as well.

There are two excellent packages available for determining code coverage: coverage.py and pytest-cov.

I recommend using pytest-cov based on its seamless integration with pytest. It's built on top of coverage.py, from Ned Batchelder, which is the standard in code coverage for Python.

Running pytest when checking for code coverage requires the --cov argument to indicate which Python package (project in the Flask project structure) to check the coverage of:

Even when checking code coverage, arguments can still be passed to pytest:

Conclusion

This article provides a guide for testing Flask applications, focusing on:

Web Python Runner

• Why you should write tests
• What you should test
• How to write unit and functional tests
• How to run tests using pytest
• How to create fixtures to initialize the state for test functions

Python Compilers Idle

If you're interested in learning more about Flask, check out my course on how to build, test, and deploy a Flask application: