Skip to content

ENGINEERING.md

Latest commit

 

History

History
183 lines (147 loc) · 13.1 KB

ENGINEERING.md

File metadata and controls

183 lines (147 loc) · 13.1 KB
Raw

Engineering standards

Companion to docs/DECISIONS.md, which is frozen and defines what to build. This document defines how well it has to be built. It is not frozen: it tightens, never loosens. The reasoning behind it is ADR-0008.

Phases 0 and 1 shipped correct measurements through code that would not survive a production review. Five declared dependencies that nothing imported, exec(compile(...)) in place of import, a fetcher with no retry against APIs that throttle, decision thresholds defined in two places, and a measurement whose sample was drawn without a seed. The numbers were right; the engineering was not. Only the measurement gate existed, so only the measurement got checked.

The standard

A phase is not complete when its measurement runs. It is complete when its code would survive being paged at 3am.

Both gates must pass. A valid number produced by code that fails the engineering gate is not a finished phase and may not be tagged.

This is not licence to gold-plate. The architecture discipline still binds: no component enters without a measurement that earns it. Production quality means the code that exists is correct, observable and operable, not that more code exists.

Part I — The universal bar

Every phase, every file, no exceptions.

1. Dependencies are real or absent

  • Every entry in pyproject.toml is imported by code that ships. An unused dependency is an unowned supply-chain risk and a false signal of sophistication.
  • Every new dependency is justified in its commit message against the ladder: stdlib → already-installed → new dep. Prefer stdlib.
  • uv.lock committed. Added via uv add, never by hand-editing pyproject.toml.

2. Module boundaries are ordinary Python

  • Configuration is imported. exec(compile(path.read_text()), globals()) is banned — it defeats static analysis, type checking, linting, and IDE navigation, and injects arbitrary names into module globals.
  • No sibling-directory sys.path mutation. The package is installed (uv sync) and imported by name.

3. Every constant has exactly one home

  • Business assumptions → config.py, labelled ASSUMED, per CLAUDE.md.
  • Policy thresholds that a decision depends on → config.py. A threshold embedded in a function body is an assumption hiding from tools/check_claims.py.
  • Only local, structural constants (buffer sizes, retry counts) may live in-module, and they are named, not literal, at the point of use.

4. I/O is hostile until proven otherwise

Any code that crosses a process boundary — network, disk, subprocess — must handle:

  • Timeouts. Explicit, never the library default.
  • Retry with backoff and jitter, with a bounded attempt count and a bounded total budget. Retry only idempotent operations and only on retryable classes.
  • Rate limiting / 429 / Retry-After. NVD and CISA both throttle.
  • Partial and corrupt responses. Digest-verify before use; a truncated .csv.gz must fail loudly, not silently parse to zero rows.
  • Resumability. A fetch interrupted at file 200 of 246 restarts from 200, not 0.

A silent except: pass, a bare except Exception without re-raise or structured log, and a swallowed parse error that yields an empty result are each an automatic gate failure. The M3A parser's try: float(...) except ValueError: continue is the pattern to avoid: it cannot distinguish "this row is a header" from "this file is corrupt".

5. Types and lint are enforced, not aspirational

  • ruff check and ruff format --check clean. No blanket # noqa.
  • mypy --strict clean on src/ and tools/. scripts/measure/ may run non-strict, but every public function is annotated.
  • No Any at a module boundary without a comment naming why.

6. Failure is legible

  • Structured logging (stdlib logging, JSON formatter) — not print. Every measurement and every system action logs: what it read, how many records, what it skipped and why.
  • Skip counts are output, not silence. Any loop that drops records emits the count and reason into results/<id>.json. A measurement that silently discards 40% of its input is a fabricated number under a different name.
  • Errors carry context: which file, which line, which CVE.

7. Tests test behaviour, not shape

  • Every branch of a decision path has a test. Every parser has a malformed-input test.
  • Every I/O path has a failure-injection test: timeout, 429, truncated body, corrupt digest.
  • Property tests (hypothesis) where the invariant is stateable — e.g. policy output is always a member of the action set; loss is non-negative and monotone in effort.
  • No network in tests. Fixtures are recorded bytes committed under tests/fixtures/.
  • A test that asserts a function returns something is not a test.

8. Determinism and reproducibility

  • Same corpus + same code + same config → byte-identical results/<id>.json.
  • Every results JSON carries: run ID, UTC timestamp, git commit SHA, corpus manifest digest, and the config values it consumed. A result that cannot name its inputs cannot be reproduced and therefore is not evidence.
  • No wall-clock or RNG in a measurement path without a seed recorded in the output.

9. Security is not deferred to Phase 7

  • Credentials only in adapters, only from environment or a secrets file, never in source, never in results/, never in logs.
  • All external input — advisory text, CVE descriptions, package metadata — is untrusted. It is never eval'd, never interpolated into a shell command, never passed to a subprocess without an argument list.
  • Subprocess calls use argument lists, never shell=True.

10. The diff is reviewable

  • One concern per commit. Measurement code and its results/ JSON in the same commit (CLAUDE.md rule 3 — unchanged).
  • No dead code, no commented-out blocks, no TODO without an owner and a phase.
  • A deliberate simplification carries a comment naming its ceiling and its upgrade path — what breaks at scale, and what replaces it. A simplification without a named ceiling is indistinguishable from a defect, both to a reviewer and to whoever is paged.

Part II — Definition of done, per phase

A phase may be tagged only when all four columns are satisfied.

Phase Measurement gate (from DECISIONS.md, unchanged) Engineering gate Operability gate Adversarial gate
0 — Data thesis M1/M2 force re-evaluation architecture Fetchers retry, resume, rate-limit, digest-verify, and report skip counts make fetch is idempotent and restartable from any interruption Corpus rebuilt from manifest on a clean checkout reproduces every digest
1 — Decision core Action-set gain measured and reported Thresholds in config.py; policy is total over the evidence space; loss function property-tested Every decision emits a decision record naming rule, inputs and config version Policy is exhaustive: no evidence combination returns None or falls through
2 — Executable plane Change-failure rate on ≥100 green-baseline repos Runner is a typed adapter with explicit resource limits; no credential reachable from sandbox Runner emits per-run resource usage, exit class and truncation flags Escape suite: network egress, host FS, host socket, fork bomb, and OOM each provably contained (see docs/SANDBOX.md)
3 — Durability Replay reproduces an earlier-epoch decision bit-for-bit Checkpoint schema versioned with a forward-compat test; leases fenced with monotonic tokens Resume path logs epoch, lease, fence token and evidence digest Kill-at-every-boundary chaos: process killed between every pair of checkpoints; no state corrupted, no decision lost
4 — Effects & authority Duplicate-effect count = 0 under injected timeouts Idempotency key derivation is pure, deterministic and unit-tested; ledger append-only with integrity check Ledger queryable by run, action and effect state; UNKNOWN effects surfaced, never auto-resolved Injected: timeout-after-effect, duplicate delivery, out-of-order ack, approval replay after policy change
5 — Evidence & adversarial Advisory text with instructions changes 0 policy decisions Untrusted text never reaches a code path with authority; supersession is a pure function over a total order Authority ranking and supersession decisions are logged with the losing source named Injection corpus with attacker-authored advisory text; measured, not asserted. Includes indirect injection via CVE references and README content
6 — Topology Split retained only on measured gain No shared mutable state between agents; message schema versioned Per-agent traces correlate to one run ID Byzantine case: one agent returns adversarial output; system does not act on it
7 — Production hardening Chaos suite passes (worker kill, provider timeout, stale evidence, duplicate delivery) Every item in the DECISIONS.md Phase 7 list has a test that fails when the mechanism is removed SLOs measured, not declared; cost ledger reconciles to provider billing units Full chaos matrix run to a documented steady-state, not a single pass
8 — Publish Every headline number traces to config.py or corpus make verify green from a clean clone with no local state README reproduces every result from scratch on a fresh machine An external reader can falsify any headline claim using only what is committed

"Fails when the mechanism is removed" is the standard for Phase 7 and a good standard everywhere: a test that still passes after you delete the code it tests is decoration.

Part III — The mechanical gate

Standards not enforced by CI are wishes. make verify is extended to enforce Parts I–II where enforcement is mechanically possible:

make verify  =  lint  +  format-check  +  typecheck  +  test (with coverage floor)
                +  check_claims.py  +  check_engineering.py

tools/check_engineering.py enforces the subset that is statically decidable:

  1. Every pyproject.toml dependency is imported somewhere under src/, scripts/ or tools/.
  2. No exec(/eval( outside an explicitly allowlisted file (currently: none).
  3. No shell=True, no bare except:, no except Exception: pass.
  4. No float/int literal thresholds compared against evidence fields outside config.py.
  5. Every results/*.json contains run_id, git_sha, manifest_digest, config_used.

Coverage floor: 90% on src/ (the decision plane — every branch is a decision that someone's production dependency graph depends on) and 70% on scripts/measure/ (parsers and aggregation logic; the corpus itself is the integration test). These are declared engineering standards, not measurements — they are policy, and they ratchet up, never down.

Part IV — Remediation (phases 0–1, closed)

Phases 0 and 1 were tagged under the measurement gate alone. Their measurements stand — nothing below changes a number in results/. Their code was brought up to the engineering bar before Phase 2 was tagged. All items are closed.

# Debt Bar violated Status
R1 requests, pandas, pyarrow, python-dateutil, packageurl-python declared, none imported anywhere I.1 Closed. All five removed; project is pure stdlib.
R2 exec(compile(CONFIG_PY.read_text()), globals()) in scripts/fetch/all.py and every scripts/measure/*.py I.2 Closed. All scripts now import config; enforced by check_engineering.py.
R3 fetch() has a timeout but no retry, no backoff, no 429 handling, no resume I.4 Closed. src/io_utils.py — bounded retry, exponential backoff + jitter, Retry-After, digest verify, gzip integrity, resume.
R4 Policy thresholds hardcoded in src/policy.py; duplicated in m3a_*.py I.3 Closed. Single definition in config.py; both call sites import.
R5 Parsers silently continue past unparseable rows; skip counts never reported I.4, I.6 Closed. src/parsers.py — SkipReport tracks every drop with reason.
R6 No structured logging; progress via print I.6 Closed. src/log.py — stdlib logging with JSON formatter.
R7 No mypy; no ruff format check; lint not part of verify I.5, III Closed. mypy + pytest-cov + hypothesis added; make verify runs all gates.
R8 results/*.json do not record git SHA, manifest digest or config consumed I.8 Closed. src/results.py — write_result() injects provenance fields.
R9 No failure-injection tests and no malformed-input tests I.7 Closed. 48 tests: timeout, 429, truncated gzip, corrupt JSON, malformed rows, property tests.
R10 zip(..., strict=False) in loss.py I.7 Closed. strict=True.
R11 cost_weights.get("request_exception", 5) — unlabelled default taxonomy, I.3 Closed. Missing key raises KeyError.
R12 Unseeded random.sample in M1 I.8 Closed. Seed in config.py::SAMPLE_SEED.

Gate: R1–R12 closed. make verify green (lint + format + mypy --strict + 120 tests at 99.19% coverage + check_claims + check_engineering).