Skip to content

TESTING.md

Latest commit

 

History

History
51 lines (30 loc) · 4.68 KB

TESTING.md

File metadata and controls

51 lines (30 loc) · 4.68 KB
Raw

DenCrypter Testing in Detail

Overview

In order to test DenCrypter, I chose to use PyTest.

Each module of the program has a dedicated test file, with the suite totaling 158 tests as of writing. The test organisation is detailed in the corresponding section.

The latest test run, which was made for version 2.0 as of writing, shows 100% pytest-cov coverage with justified exclusions.

Tests can be run following the steps outlined in the dedicated section.

Testing philosophy

The test suite was made with one core rule: to verify the program behaves correctly, instead of chasing line coverage.

Typical examples include checking errors are emitted when providing malformed inputs or detecting encrypted file tampering, as well as ensuring round-trip integrity per cipher.

The resulting coverage is merely a consequence, not the target or purpose of the suite. The previously mentioned 100% coverage figure therefore reflects behavioural coverage; some lines are defensive but unreachable, and therefore serve no purpose for behavioural testing. They were marked with a # pragma: no cover annotation/comment in the code.

Test organisation

Tests are located in the tests directory at the root of the repo. Each module of the program has a dedicated test file:

  • test_cli.py: covers argument parsing (including --force tests where applicable, and invalid/missing arguments or combinations thereof), end-to-end main() behaviour over all three operations, as well as error handling with proper exit messages and error codes, including guards for cases argparse doesn't handle
  • test_crypto.py: covers testing of text encryption/decryption helpers (valid args, or invalid ones e.g. inappropriate cipher or tampered args), key generation happy path and failure modes (invalid key size, nonexistent parent path, already existent file), and full file encryption/decryption methods (similar arg checks as the helpers, plus file-level header verification)
  • test_format.py: covers header construction and invalid key-cipher pairings (happy path + invalid cipher), header parsing (invalid header size or fields), with match=[...] assertions to ensure the proper error messages are emitted
  • test_inputs.py: covers input validation checks, including valid key-cipher pairings (detailed in constants.py) and invalid ones (swapped matchings, empty entries or unknown cipher), key path validation (.key extension, path existence, ensuring it is not None), output path validation (with and without --force), and key loading (valid hex round-trip, and rejection of invalid hex)
  • test_e2e.py: end-to-end DenCrypter test of key generation -> encryption -> decryption workflow, using subprocess.run with the installed dencrypter CLI entry point; verifies proper return codes, file creation/existence, stdout success text, whether ciphertext differs from plaintext, and round-trip byte content equality

Notes

  • For simplicity, decryption tests use round-trip decryption instead of Known-Answer Test (KAT) vectors. Ideally, one would use KATs to properly validate decryption, but doing so is difficult in the case of DenCrypter as it uses a custom encryption header.
  • Some files do not cover or re-test failure modes covered in other files, e.g. inputs.py function failures are not re-tested in test_crypto.py; doing so would result in redundant code.

Coverage exclusions

Some lines of code were left untested; these are indicated by # pragma: no cover annotations, and justified in code.

As of writing, there are two regions containing such annotations:

  • cli.py: the call to main() in the if __name__ == "__main__" block, which runs only when the module is executed directly (and not when imported by tests, which makes testing it meaningless)
  • crypto.py: two defensive try-except blocks for build_header() and encrypt_text(); these would fail only if the provided key or cipher is/are invalid. But the two fields are verified before reaching these blocks. Leaving them could seem redundant, but I believe this makes for good defence-in-depth, since any alteration/tampering of the already-validated key or cipher would be caught and propagated as a DencrypterError subclass, which is the error type expected in cli.py. Testing these lines would require mocking such tampering, which would be tantamount to testing a mock instead of the expected behaviour of the program.

Running the tests

To run the tests, from your shell:

  1. Install dev dependencies: pip install -e ".[dev]"
  2. Run the test suite: pytest
  3. Run the test suite with coverage: pytest --cov=dencrypter --cov-report=term-missing