Writing, running, and testing code¶

Fixing a bug or implementing a feature will require you to write some new code.

To begin working on code, ensure you have a development environment set up, and you are working on a branch

We have a code style guide that outlines our guidelines for writing code for BeeWare.

Test-driven development¶

A good way to ensure your code is going to do what you expect it to, is to first write a test case to test for it. This test case should fail initially, as the code it is testing for is not yet present. You can then write the code changes needed to make the test pass, and know that what you've written is solving the problem you are expecting it to.

Run your code¶

Once your code is written, you need to ensure it runs. You'll need to manually run your code to verify it is doing what you expect. If you haven't already, you'll want to write a test case for your changes; as mentioned above, this test should fail if your code is commented out or not present.

You'll add your test case to the test suite, so it can be run alongside the other tests. The next step is to run the test suite.

Running tests and coverage¶

Briefcase uses tox to manage the testing process and pytest for its own test suite.

The default tox command includes running:

pre-commit hooks
towncrier release note check
documentation linting
test suite for available Python versions
code coverage reporting

This is essentially what is run by CI when you submit a pull request.

To run the full test suite, run:

macOSLinuxWindows

(.venv) $ tox

(.venv) $ tox

(.venv) C:\...>tox

The full test suite can take a while to run. You can speed it up considerably by running tox in parallel, by running tox p (or tox run-parallel). When you run the test suite in parallel, you'll get less feedback on the progress of the test suite as it runs, but you'll still get a summary of any problems found at the end of the test run. You should get some output indicating that tests have been run. You may see SKIPPED tests, but shouldn't ever get any FAIL or ERROR test results. We run our full test suite before merging every patch. If that process discovers any problems, we don't merge the patch. If you do find a test error or failure, either there's something odd in your test environment, or you've found an edge case that we haven't seen before - either way, let us know!

In addition to the tests passing, this should report 100% test coverage.

Running test variations¶

Run tests for multiple versions of Python¶

By default, many of the tox commands will attempt to run the test suite multiple times, once for each Python version supported by Briefcase. To do this, though, each of the Python versions must be installed on your machine and available to tox's Python discovery process. In general, if a version of Python is available via PATH, then tox should be able to find and use it.

Run only the test suite¶

If you're rapidly iterating on a new feature, you don't need to run the full test suite; you can run only the unit tests. To do this, run:

macOSLinuxWindows

(.venv) $ tox -e py

(.venv) $ tox -e py

(.venv) C:\...>tox -e py

Run a subset of tests¶

By default, tox will run all tests in the unit test suite. When you're developing your new test, it may be helpful to run just that one test. To do this, you can pass in any pytest specifier as an argument to tox. These test paths are relative to the briefcase directory. For example, to run only the tests in a single file, run:

macOSLinuxWindows

(.venv) $ tox -e py -- tests/path_to_test_file/test_some_test.py

(.venv) $ tox -e py -- tests/path_to_test_file/test_some_test.py

(.venv) C:\...>tox -e py -- tests/path_to_test_file/test_some_test.py

You'll still get a coverage report when running a part of the test suite - but the coverage results will only report the lines of code that were executed by the specific tests you ran.

Run the test suite for a specific Python version¶

By default tox -e py will run using whatever interpreter resolves as python on your machine. If you have multiple Python versions installed, and want to test a specific Python version from the versions you have installed, you can specify a specific Python version to use. For example, to run the test suite on Python 3.10, run:

macOSLinuxWindows

(.venv) $ tox -e py310

(.venv) $ tox -e py310

(.venv) C:\...>tox -e py310

A subset of tests can be run by adding -- and a test specification to the command line.

Run the test suite without coverage (fast)¶

By default, tox will run the pytest suite in single threaded mode. You can speed up the execution of the test suite by running the test suite in parallel. This mode does not produce coverage files due to complexities in capturing coverage within spawned processes. To run a single python version in "fast" mode, run:

macOSLinuxWindows

(.venv) $ tox -e py-fast

(.venv) $ tox -e py-fast

(.venv) C:\...>tox -e py-fast

A subset of tests can be run by adding -- and a test specification to the command line; a specific Python version can be used by adding the version to the test target (e.g., py310-fast to run fast on Python 3.10).

Code coverage¶

Briefcase maintains 100% branch coverage in its codebase. When you add or modify code in the project, you must add test code to ensure coverage of any changes you make.

However, Briefcase targets multiple platforms, as well as multiple versions of Python, so full coverage cannot be verified on a single platform and Python version. To accommodate this, several conditional coverage rules are defined in the tool.coverage.coverage_conditional_plugin.rules section of pyproject.toml (e.g., no-cover-if-is-windows can be used to flag a block of code that won't be executed when running the test suite on Windows). These rules are used to identify sections of code that are only covered on particular platforms or Python versions.

Of note, coverage reporting across Python versions can be a bit quirky. For instance, if coverage files are produced using one version of Python but coverage reporting is done on another, the report may include false positives for missed branches. Because of this, coverage reporting should always use the oldest version Python used to produce the coverage files.

Understanding coverage results¶

At the end of the coverage test output there should be a report of the coverage data that was gathered:

 Name    Stmts   Miss Branch BrPart   Cover   Missing
 ---------------------------------------------------
 TOTAL    7540      0   1040      0  100.0%

This tells us that the test suite has executed every possible branching path in the code. This isn't a 100% guarantee that there are no bugs, but it does mean that we're exercising every line of code in the codebase.

If you make changes to the codebase, it's possible you'll introduce a gap in this coverage. When this happens, the coverage report will tell you which lines aren't being executed. For example, lets say we made a change to some/interesting_file.py, adding some new logic. The coverage report might look something like:

 Name                                 Stmts   Miss Branch BrPart  Cover   Missing
 -------------------------------------------------------------------------------
 src/some/interesting_file.py           111      1     26      0  98.1%   170, 302-307, 320->335
 -------------------------------------------------------------------------------
 TOTAL                                 7540      1   1726      0  99.9%

This tells us that line 170, lines 302-307, and a branch jumping from line 320 to line 335, are not being executed by the test suite. You'll need to add new tests (or modify an existing test) to restore this coverage.

Coverage report for host platform and Python version¶

You can generate a coverage report for your platform and version of Python. For example, to run the test suite and generate a coverage report on Python 3.10, run:

macOSLinuxWindows

(.venv) $ tox -m test310

(.venv) $ tox -m test310

(.venv) C:\...>tox -m test310

Coverage report for host platform¶

If all supported versions of Python are available to tox, then coverage for the host platform can be reported by running:

macOSLinuxWindows

(.venv) $ tox p -m test-platform

(.venv) $ tox p -m test-platform

(.venv) C:\...>tox p -m test-platform

Coverage reporting in HTML¶

A HTML coverage report can be generated by appending -html to any of the coverage tox environment names, for instance:

macOSLinuxWindows

(.venv) $ tox -e coverage-platform-html

(.venv) $ tox -e coverage-platform-html

(.venv) C:\...>tox -e coverage-platform-html

Changing Briefcase templates¶

Briefcase uses templates at runtime to generate the files and project structure for applications. When a contributor runs commands such as briefcase new or briefcase create, Briefcase selects and renders one or more templates to produce platform-specific project scaffolding.

These templates are maintained separately from Briefcase's core code, which allows template development and testing to evolve independently of Briefcase releases.

Broadly, Briefcase works with two categories of templates:

Wizard templates These templates are used during briefcase new to generate the initial project layout, including files such as pyproject.toml, application metadata, and basic source structure.
Platform-specific templates These templates are used during briefcase create and later stages to generate platform-dependent project files (for example, to produce a macOS app or iOS Xcode project). Each platform has its own template repository that defines how the application is structured and built on that platform.

When contributing changes to templates, you need to be able to test those changes locally. Briefcase has features to make this testing possible.

Testing wizard template changes¶

Wizard templates are used when creating a new project with briefcase new. Briefcase allows a user to specify a custom wizard template repository and branch when creating a new project. This makes it possible to test template changes without modifying Briefcase's core code.

To test a modified wizard template:

Fork and clone the wizard template repository you are working on.
Make your changes in a local branch.
Run briefcase new, specifying the custom template repository and branch:
```
briefcase new --template ../../../path/to/briefcase-template
```
Alternatively, you can push your changes to GitHub, and then point Briefcase at the GitHub repository and branch for your template fork:
```
briefcase new --template "https://github.com/<your-github-username>/briefcase-template" --template-branch my-template-changes
```

Testing platform template changes¶

Platform-specific templates are used when deploying a project to a specific platform. This will usually happen when you invoke briefcase create, but it can also happen as a result of calling briefcase build, briefcase run, or briefcase package if briefcase create hasn't been invoked. These templates define how an application is structured, configured, and built for a specific platform.

Briefcase supports using a local copy of a template using the template and template_branch options. To test a modified platform template:

Fork and clone the platform template repository you are working on (e.g., this template for macOS apps).
Make your changes in a local branch.
Open the project’s pyproject.toml file.
In the platform-specific configuration section, set the template to use when creating the app. During initial development, it will be easiest to point at a local checkout of the template:
```
[tool.briefcase.<platform>]
template = "../../../path/to/briefcase-<platform>-app-template"
```
However, you can can also push your changes to your GitHub repository, and configure your Briefcase project to point at that repository and branch:
```
[tool.briefcase.<platform>]
template = "https://github.com/<your-github-username>/briefcase-<platform>-app-template"
template_branch = "my-template-changes"
```

Template changes that require Briefcase changes¶

In some cases, changes to a template may depend on corresponding changes in Briefcase itself. For example, if you add a new context variable, or modify the behavior of Briefcase to change how a template variable is constructed, you may require both a Briefcase and template change.

This situation then causes problems during testing, because Briefcase's tests won't pass with an old template, and tests for the template won't pass with an old version of Briefcase.

To work around this situation, pull requests on Briefcase's template repositories support a syntax for pointing at a specific version of Briefcase. Once you've made changes locally to both Briefcase and the affected template, and you're satisfied that they work, push the changes for both to Briefcase, and open a pull request on each repository.

In the PR description for Briefcase, provide details of the template branch that is required for testing. If the template changes are needed for the Briefcase tests to pass, flag this fact in the PR description; this lets reviewers know that the process of merging the PR will be more complicated.

In the PR description for the template, include the following two lines:

briefcase-repo: https://github.com/<your-github-username>/briefcase
briefcase-ref: my-template-changes

This directs the template tests to use a branch of Briefcase for testing. This should allow the template tests to pass. The person reviewing your PR will review both the Briefcase and template changes together. If approved, the template change will be merged, and tests on Briefcase will then be re-run. This provides a final confirmation that the template and Briefcase are compatible.

It's not just about writing tests!¶

Although we ensure that we test all of our code, the task isn't just about maintaining that level of testing. Part of the task is to audit the code as you go. You could write a comprehensive set of tests for a concrete life jacket… but a concrete life jacket would still be useless for the purpose it was intended!

As you develop tests, you should be checking that the core module is internally consistent as well. If you notice any method names that aren't internally consistent (e.g., something called on_select in one module, but called on_selected in another), or where the data isn't being handled consistently, flag it and bring it to our attention by raising a ticket. Or, if you're confident that you know what needs to be done, create a pull request that fixes the problem you've found.

Once you have everything working, you can submit a pull request with your changes.