Skip to content

testing.md

Latest commit

 

History

History
604 lines (488 loc) · 27.6 KB

testing.md

File metadata and controls

604 lines (488 loc) · 27.6 KB
Raw

Testing

This project writes to real power hardware, so tests are the main guard against regressions in control logic, write gates and safety reconciliation. All the checks below run offline — no Home Assistant, Shelly, Zendure devices, InfluxDB, secrets, or network access are required.

See also developer-setup.md for the source checkout and environment, and ci-release.md for how these run in CI.

Development approach

This project develops complex and risky features contract-first with test-driven development: for control logic, write gates, safety reconciliation, config validation and the Admin apply/credential paths, a failing contract test is written and shown to fail before the production change is made. Bugs get a reproducing test before the fix. Small UI tweaks and exploratory prototypes may stay pragmatic and gain tests as they settle. This is a working policy for the areas that can write to real hardware, not a claim that every historical line of the project was produced under strict test-driven development. The canonical project requirement and scope are defined in agent-rules.md.

Before a broad run: scratch space

A full run writes tens of GB of temporary data — appliance packages, disk images, database snapshots. Point TMPDIR at a filesystem with room:

export TMPDIR=/path/with/room

tests/conftest.py refuses a broad selection that does not fit and names the fix, because a temporary filesystem that fills part-way through kills the run with an internal error rather than a test failure, and leaves the host unusable until it is cleaned. A targeted tier is never refused — the check runs after marker deselection, so it judges what will actually run. EMS_ALLOW_SMALL_SCRATCH=1 overrides it.

Quick local loop

Compile check (run after any change to the entry script, ems/, emsctl.py or scripts/check_log_events.py):

python3 -m py_compile ems-solarflow-api-control.py ems/*.py emsctl.py scripts/check_log_events.py

Self-test:

python3 -B ems-solarflow-api-control.py --self-test

Simulation (no hardware or network):

python3 -B ems-solarflow-api-control.py --simulate --max-cycles 1

Replay a captured trace:

python3 -B ems-solarflow-api-control.py --replay /path/to/trace.jsonl --once

Test selection

The suite is large — over 8000 Python tests and 47 Playwright specs — and the full non-Docker run takes about 16 minutes on a developer machine against about two for the fast tier. Run the tier that matches your change; the full suite is a release gate, not a per-edit loop.

Two marker dimensions

Every test module carries exactly one execution level and any number of functional areas. The dimensions are independent and overlap on purpose.

Level Meaning
unit isolated deterministic unit tests
contract API, serialization, schema and frontend source contracts
integration multiple production components without a real browser journey
e2e complete user or service journeys
Execution property Meaning
docker requires a Docker daemon or Docker-first environment
browser requires Playwright or a browser runtime
slow exceeds the normal pull-request runtime budget
Functional area Meaning
admin Admin Server and Admin UI behavior
setup Guided Setup lifecycle and installation flow
maintenance Maintenance workflows
workflow workflow ownership and lifecycle
authority Device Plan, Preview, Apply and mutation authority
config configuration generation, validation and mutation
mqtt local MQTT, Zendure MQTT and broker workflows
power_control power allocation and output safety
backup_restore backup, restore and recovery workflows
system_build System Build and deployment transitions
documentation documentation, licensing and third-party inventory contracts
appliance Raspberry Pi Appliance Manager, its image and its packaging

simulation, regression and mqtt_release stay registered for the existing gates. Every marker lives in pytest.ini and --strict-markers is enabled, so a typo fails the run instead of silently selecting nothing.

Fast developer tier

./scripts/test-fast.sh
# pytest -q -m "(unit or contract) and not docker and not browser and not slow" --maxfail=1

Targeted functional tiers

./scripts/test-admin.sh authority
./scripts/test-admin.sh setup
./scripts/test-admin.sh maintenance
./scripts/test-admin.sh workflow
./scripts/test-mqtt.sh
./scripts/test-mqtt.sh integration

The same selections without the wrappers:

pytest -q -m "admin and authority and not slow"
pytest -q -m "admin and setup and not slow"
pytest -q -m "mqtt and not docker and not slow"
pytest -q -m "power_control and not slow"
pytest -q -m "system_build and not docker and not slow"
pytest -q -m "backup_restore"
pytest -q -m "docker"
pytest -q -m "documentation"

Pull-request groups

Each group runs independently and mirrors one CI job:

./scripts/test-pr.sh core
./scripts/test-pr.sh appliance
./scripts/test-pr.sh admin
./scripts/test-pr.sh mqtt
./scripts/test-pr.sh power-control
./scripts/test-pr.sh docker
./scripts/test-pr.sh chromium-critical
./scripts/test-pr.sh firefox-smoke

The six groups are an exact partition: every collected test runs in exactly one of them, and none runs twice. Functional markers stay overlapping descriptions of behavior — a module may be both admin and mqtt. Only execution ownership is exclusive, resolved by a fixed priority:

docker > power-control > mqtt > admin > appliance > core
Group Marker expression
power-control power_control and not docker
mqtt mqtt and not power_control and not docker
admin admin and not mqtt and not power_control and not docker
appliance appliance and not admin and not mqtt and not power_control and not docker
core not appliance and not admin and not mqtt and not power_control and not docker
docker docker

tests/test_test_classification.py proves both directions: the union equals the full collection, every pair of Python groups is disjoint, and each non-Docker test has exactly one owner. To run a functional area rather than a CI group, use the plain marker (pytest -m "admin and authority") — that selection is intentionally overlapping.

On the merge

There is no nightly schedule. .github/workflows/simulated-regression-tests.yml and .github/workflows/playwright-e2e.yml run the full non-Docker suite on both supported Python versions, the strict deprecation check and the complete Chromium and Firefox Admin suites on the push to main, skipping them on pull requests. The Docker-first and System Build tiers run on every pull request already. The Admin upgrade/recovery journey belongs to .github/workflows/admin-replacement-canary.yml.

Release candidate

./scripts/test-rc.sh          # every gate, in order
./scripts/test-rc.sh --list   # print the gate list

Gates: static checks, the full non-Docker Python suite, the simulation and power_control gate, the authority regressions, the security regressions, the System Build tier, the Docker-first tier, the full Chromium and Firefox Admin suites, the Admin replacement/recovery suite, the Appliance Manager browser suite, the Dashboard cockpit suite, the generated config template and a clean-working-tree check. ./scripts/test-rc.sh --list prints them, and that list is the authority. The RC tier never deselects a known failure.

Playwright groups

The specs carry Playwright tags, so one configuration serves every group:

npx playwright test --project=chromium --grep @smoke
npx playwright test --project=chromium --grep @authority
npx playwright test --project=firefox --grep @smoke
npx playwright test --project=chromium --grep "@setup|@authority"
npx playwright test --project=chromium               # full Admin suite

Tags: @smoke (fast critical journeys), @setup, @maintenance, @authority, @workflow, @system-build.

Documentation screenshots

./scripts/capture-docs-screenshots.sh            # Admin + Dashboard pages
npx playwright test --config=playwright.appliance.config.ts capture-docs \
    # with EMS_APPLIANCE_CAPTURE_DOCS=1        # Appliance Manager pages
python3 scripts/capture_appearance_gallery.py    # the appearance strips

The first two photograph the product's pages. The third photographs one page three times with a single appearance axis changed -- palette, object style, density -- because a choice whose entire effect is how something looks cannot be documented in prose. All three use the same deterministic servers as the browser suites, so no capture contains a real host, address, serial or key.

Each screenshot directory carries a README.md listing every image, and contract tests refuse a capture that is not listed or a listing without an image. Two writers now share the Admin and Appliance directories, so those manifests are a union; add a row when you add a capture.

The other two surfaces

Three deployables, three origins, three browser suites. They share nothing -- not a config, not a server, not a port -- because the products do not share anything either.

npx playwright test --config=playwright.appliance.config.ts   # Appliance Manager
npx playwright test --config=playwright.dashboard.config.ts   # EMS Dashboard

Each brings its own deterministic server: the Manager's is the scripted EMS_APPLIANCE_TEST_MODE host, the cockpit's is scripts/serve_dashboard_preview.py, which serves the real dashboard/static/ assets with synthetic API payloads -- no hardware, no MQTT, no history database, no secrets.

Four things are worth knowing before adding a spec to either:

  • Measure with a real exit code. npx playwright test | tail -20 reports the exit status of tail, which is always zero, and the N failed line sits above the window. Write to a log and read $? straight after.
  • WebKit is not installed everywhere. The default config lists it, so locally pass --project=chromium --project=firefox or collect a few hundred "Executable doesn't exist" failures.
  • The cockpit's network is never idle. /api/events is an open SSE stream, so waitUntil: "networkidle" sits out its timeout on every navigation. Wait for an element.
  • Do not park a page with window.scrollTo. In Firefox a focus() that had to bring its target on screen leaves a scroll-into-view pending, and that pending scroll is applied on top of the next programmatic scroll one frame later. A test that scrolls and writes down window.scrollY records a position the page left fifteen milliseconds afterwards, and then fails somewhere else entirely -- the appliance case was named after a two-second poll that had not run yet. A reflow clears the pending scroll, so a heavier page hides it and a lighter one exposes it, which makes it look like a flake and look like it belongs to whatever last touched the CSS. Real input is not undone in either engine: press a key. tests/e2e-appliance/helpers.ts has parkAtBottom, which presses End and then waits for the page to actually arrive instead of assuming it did.

Prerequisites

  • Python tiers need only the project virtualenv.
  • docker tiers need a reachable Docker daemon (docker info). Without one the Docker suites skip with a precise reason instead of failing.
  • Playwright tiers need npm ci and npx playwright install chromium firefox. No tier installs dependencies.
  • The RC admin-replacement gate replaces one published Admin container with another, both pinned by digest, so it needs ADMIN_REPLACEMENT_RUNTIME, ADMIN_REPLACEMENT_EVENTS, the source identity (CANARY_SOURCE_TAG, CANARY_SOURCE_REVISION, CANARY_SOURCE_BUILD_ID, CANARY_SOURCE_ADMIN_DIGEST) and the target identity (CANARY_TAG, CANARY_REVISION, CANARY_BUILD_ID, CANARY_ADMIN_DIGEST, CANARY_EMS_DIGEST). Both sides come from the Development catalogue; python3 scripts/resolve_canary_builds.py --catalogue <file> prints them and ./scripts/test-rc.sh names the missing ones before running any gate.

Classifying a new test

Declare the markers at module level — that declaration is the authority, not the file name:

pytestmark = [
    pytest.mark.admin,
    pytest.mark.authority,
    pytest.mark.integration,
]

Rules:

  • Exactly one level marker per module (unit, contract, integration, e2e). tests/test_test_classification.py fails when a module declares none or several.
  • Add every functional area that applies; overlap is intended.
  • Use per-test markers only when one module genuinely mixes categories.
  • Use slow when a test exceeds the pull-request budget, so the fast tier and the PR groups stay usable.
  • The mqtt_release allowlist in tests/conftest.py is the one remaining name-based bridge and is guarded by a contract test. Classify anything new with markers instead.

Required regression check

The offline power-control regression tests are the deterministic CI gate. They must stay green:

pytest tests/ -m "simulation and power_control"

Debugging selection

pytest --markers                                    # every registered marker
pytest --collect-only -q -m "admin and authority"
pytest --collect-only -q tests/test_write_gates.py
pytest -k "stale_device_plan"                       # by test-name substring
pytest tests/test_write_gates.py::test_gate_blocks_write

An empty selection exits with code 5. The tier scripts turn that into an explicit error instead of a silent pass.

Log validation

python3 scripts/check_log_events.py /tmp/ems-sim.log \
  --require startup \
  --require target_calculation

Documentation and contract tests

Several tests protect docs and public contracts rather than runtime behavior. The documentation-content ones carry the documentation marker, so pytest -q -m documentation runs the whole set:

  • tests/test_docs_user_structure.py — the user / technical / developer documentation split and README routing.
  • tests/test_docs_user_guides.py — the step-by-step guides under docs/user/admin/ and docs/user/dashboard/: required pages, the shared section shape, resolvable image and relative links, every committed screenshot embedded somewhere, pairwise-distinct screenshots, descriptive alt text, the capture manifests matching the committed files, and the no-secrets scan.
  • tests/test_docs_admin_media.py — the Admin demo videos and their static screenshot fallbacks.
  • tests/test_third_party_licenses.py — THIRD_PARTY_LICENSES.md against the requirements files, package.json, package-lock.json, the vendored static assets and the container base images, plus the negative cases that prove tools/check_third_party_licenses.py actually rejects drift.
  • tests/test_agent_rules_contract.py — the canonical rule set and supported agent entry-point links.
  • tests/test_docker_docs_contract.py, tests/test_docker_first_setup.py — the Docker Bootstrap installer/compose/docs promise.
  • tests/test_issue_templates.py — issue-template documentation links.
  • tests/test_test_classification.py — the marker registry, the documented tier selections and the pull-request group partition.
  • tests/test_ci_workflow_docker_split.py — how the CI groups are split.
  • tests/test_ci_workflow_contexts.py, tests/test_ci_workflow_commands.py — that a workflow names its contexts where they exist and runs shell that runs.

When you move or rename docs, update these tests (or the redirect stubs) so the links stay honest.

Regenerating the documentation screenshots

The user guides are screenshot-led. Regenerate every image with:

./scripts/capture-docs-screenshots.sh            # Admin + Dashboard
./scripts/capture-docs-screenshots.sh admin      # Admin Console only
./scripts/capture-docs-screenshots.sh dashboard  # EMS Dashboard only

Both capture scripts start their own loopback-only preview server from the deterministic fixtures in tests/fixtures/admin_docs/ and scripts/dashboard_preview_data.py, and shut it down again when they finish. No Docker, hardware, discovery, MQTT broker, Zendure credential, config.json or runtime state is involved, unrelated containers are untouched, and nothing is pushed. Requirements: headless firefox and ImageMagick convert.

Add a screen by extending SCREENS in scripts/capture_admin_docs.py or scripts/capture_dashboard_docs.py (plus a driver in scripts/admin_docs_preview.js for Admin), then embed it in a guide — tests/test_docs_user_guides.py enforces that the manifest, the committed files and the asset README stay in step.

Two traps worth knowing:

  • Guided Setup steps 02–05 are authorized by a server-confirmed setup transition, so the preview must serve one. Without it every setup screen silently falls back to step 01 and the captures become byte-identical duplicates; the distinctness test is what catches that.
  • The Admin driver only runs after the SPA's own authenticated workflow resume has finished, because that resume re-opens Guided Setup on step 01.

After a run, review git status --short docs/assets/screenshots before committing.

Appliance tiers that need a real machine

Three appliance claims cannot be settled by pytest, because what they assert is a property of a booted operating system, of a package manager, or of an image builder — not of a Python object. Each has a driver script; each reports PASS, FAIL or NOT RUN and never reports a skipped run as a pass.

# A packaged appliance in a Debian Trixie guest that really booted.
scripts/appliance-smoke-vm-amd64.sh [--rpi-image-gen DIR] [--keep]

# The real image builder, in a guest that is thrown away afterwards.
scripts/appliance-builder-vm.sh --profile rpi5 [--profile rpi4] --output DIR

# The strict release gate, in that same guest — the only host it can pass on.
scripts/appliance-builder-vm.sh --release-gate --profile rpi5 --profile rpi4 --output DIR

# The oldest board, which boots from SD and nothing else.
scripts/appliance-builder-vm.sh --profile rpi3 --output DIR

A release is three images, one per board. appliance-builder-vm.sh requires at least one --profile and exits 2 without one, so covering a release through the VM wrapper means naming every profile. scripts/appliance-release-gates.sh, run directly, is the one that derives the list from rpi_image_gen.HARDWARE_PROFILES when --profile is omitted.

The same build, in CI

.github/workflows/appliance-image.yml builds the images on a GitHub-hosted runner instead, on workflow_dispatch and never on a push. It installs the generator's declared dependency set on the runner, so there is no guest: the disposable VM exists to keep those packages off a developer's workstation, and a hosted runner is already disposable. It then runs the same appliance-release-gates.sh --mode builder over the same pinned generator, one job per board, and uploads the .img.xz, the build authority, the builder environment and every gate log.

What it produces is a build and not a release, and the difference is not paperwork. packaging/appliance/vm/base-images.lock.json approves exactly one builder image and tells it from every other machine by a kernel pattern no hosted runner carries, so appliance-finalize-rpi-release.sh refuses an image from CI with builder_environment_untrusted. The workflow does not work around that: it records the builder environment honestly and passes no --base-image-sha512, because a hosted runner has no approved base image to name. Flashing verifies no signature, so a CI build is enough for ../appliance/hardware-validation.md and is not enough for a signed release.

A build every board finished is published, by the workflow's fourth job, as a GitHub prerelease carrying the .img.xz and .img.xz.sha256 per board, a combined SHA256SUMS, the Manager .deb, each image's build authority and builder environment, and the gate logs as gate-evidence-<board>.zip. ../user/appliance/install.md already sent operators to the Releases page for exactly those file names; an Actions artefact expires in thirty days, sits behind a sign-in wall and lives on a run page no operator will find, so until that job existed the page described a download nobody produced.

Two properties of that job are load-bearing rather than stylistic, and tests/test_appliance_image_workflow.py asserts both. It publishes a prerelease: GitHub's /releases/latest alias skips one, and that alias is what the repository sidebar advertises and what anything asking this repository for its latest release is handed. An unsigned appliance image published as an ordinary release would stand there every week, in front of the EMS version it is not, on the page operators are sent to. And the tag is deliberately not a version — admin/releases.py lists every non-draft release of this repository as an EMS system-build target and decides eligibility by parsing the tag, so a semver-shaped tag would offer operators a build whose container images do not exist. That test runs the tag template through admin.releases.VERSION_PATTERN rather than reading it.

Signing is still refused here, and the workflow reads no repository secret at all. Publishing an unsigned build and signing one are different questions; the release page answers the first and says, in the gate runner's own words, that it is not the second.

The build runs weekly, on a cron and never on a push. Nothing in it pins a package version — trixie, trixie-updates and trixie-security are resolved as the build runs, and upstream's snapshot-pinned alternative is deliberately unused — so rebuilding is the whole mechanism by which a freshly flashed card comes up patched. tests/test_appliance_image_workflow.py holds the trap that would have made the schedule pointless: the inputs context is populated for workflow_dispatch and workflow_call and empty otherwise, so a plain inputs.publish would make every scheduled run build three images and publish nothing, as a green skip.

The Manager package, which may be signed

.github/workflows/appliance-manager-release.yml is the one workflow in this repository that reads a secret. The asymmetry is deliberate and worth stating, because it is easy to assume the builder-VM rule blocks both: an image built on a hosted runner is refused at signing time, while the .deb is reproducible from SOURCE_DATE_EPOCH and a pinned compressor, so two builds of one commit are the same bytes and an unattested builder is no objection to it.

Three jobs, so that no job holds both the signing key and permission to write to the repository, and tests/test_appliance_manager_release_workflow.py asserts exactly that. The signature is verified against packaging/appliance/config/release-keyring.gpg — the keyring the package itself installs — with gpgv, the program the appliance runs, so a key the fleet would refuse fails on a runner instead of in the field. The runbook is ../appliance/manager-releases.md.

The image build then consumes what that workflow published: scripts/appliance-fetch-manager-package.py reads the same index the fleet reads, takes the newest stable entry, verifies the signature and the digest, and hands the package to appliance-build-rpi-image.sh --manager-package. Its exit 3 means "no stable release published yet", which is a state this project is actually in and not a failure — the build then makes its own package and says so. A package that failed verification is a different matter and fails the job; tests/test_appliance_fetch_manager_package.py covers both, against real signatures rather than stubs.

The Appliance Manager .deb is built by its own job in that workflow and uploaded separately. packaging/appliance/build-deb.sh is reproducible from SOURCE_DATE_EPOCH and a pinned compressor, so two builds of one commit are the same bytes and an unattested builder is no objection to it.

tests/test_appliance_image_workflow.py keeps the workflow and rpi_image_gen.HARDWARE_PROFILES in agreement: a board added to the table that CI cannot be asked to build fails that test rather than leaving a release one artefact short.

A workflow's shell is not executed until the run that needs it, and the first dispatch of this one died on its opening line: df -PB1 --output=avail / is a combination coreutils refuses, which YAML parses and bash -n accepts. tests/test_ci_workflow_commands.py closes that class by running it — every df invocation in every workflow is executed here against a directory that exists, with each operand substituted, so the option list is the only thing under test. It also parses every shell step with bash -n, which is the cheap half and would not have caught this one.

The gate builds the images itself, so it needs the generator's prerequisites and cannot reach RESULT: PASS on a workstation that deliberately lacks them. --release-gate runs it where those prerequisites are, and brings the verdict and dist/gates/ back out. appliance-inspect-rpi-image.sh needs neither root nor a loop device: it reads the partition table, the filesystems and the files inside them straight out of the image file, which is the only way a Pi 5 root with 16 KiB ext4 blocks can be inspected on a 4 KiB-page host at all.

Both need qemu-system-x86_64, qemu-img, a writable /dev/kvm, an ISO writer (genisoimage or xorriso) and network access to cloud.debian.org. The base image is cached under $EMS_APPLIANCE_VM_CACHE and verified against the published SHA512SUMS on every run. Nothing is installed on the developer host: rpi-image-gen's dependency set — mmdebstrap, podman, uidmap, pv, btrfs-progs, dctrl-tools, python3-jsonschema, cryptsetup, flex — and the qemu-aarch64 binfmt handler are installed inside the disposable guest.

Why a container is not enough

The tier these replaced ran systemd inside a privileged container, and systemd never finished booting there. That is not a slow test, it is an absent one: a container that never builds a systemd transaction cannot disprove anything about unit ordering. Two defects lived behind it — a Requires= on a mount unit that did not exist on every host, which failed the install, and a host key generation that could never succeed on a real appliance.

What the guest tier deliberately does not claim

A generic QEMU guest is not a Raspberry Pi: it has no Pi firmware, no /proc/device-tree/chosen/bootloader, and no SD card whose partition table the first boot would grow. Faking any of that would make the verdict a statement about the fake, so those cases are reported NOT RUN and belong to ../appliance/hardware-validation.md.

Reading the build back

A build produces a build-authority.json beside the image. The inspectors (appliance-inspect-rpi-image.sh) and the strict gate (appliance-release-gates.sh) read the artefacts rather than the build log, so a build that half-succeeded is caught by the thing that reads its output, not by the thing that produced it.

What tests do not cover

Automated tests reduce risk but do not replace real hardware validation: dry-run checks, watching the first live run, and per-installation review of power and SOC limits are still required. See ../user/safety.md.

For an on-hardware measurement (not part of the offline suite) of how fast an MQTT outputLimit write reaches the inverter, see mqtt-write-latency-probe.md.