Skip to content

CLAUDE.md

Latest commit

 

History

History
210 lines (161 loc) · 9.75 KB

CLAUDE.md

File metadata and controls

210 lines (161 loc) · 9.75 KB
Raw

Sentinel — Project Constitution

You are building Sentinel: a durable vulnerability-response decisioning system for open-source software dependencies, operating under incomplete, evolving, and externally-modelled evidence.

The full rationale is in docs/DECISIONS.md. Read it before Phase 0. It is frozen. Do not redesign the use case. Architecture changes only because a measurement says so.

The engineering bar is in docs/ENGINEERING.md. Read it before writing any code. It is not frozen — it tightens, never loosens.

PRIME DIRECTIVE

Never invent a number.

This project exists because a previous draft filled its architecture with plausible, confident, fabricated figures. Every number you write must come from code that ran, or carry a label saying where it came from. A missing measurement is reported as missing. An estimate is labelled as an estimate. A fabricated number is a project-ending defect, not a rough draft.

If you find yourself about to write a figure you did not compute, stop and write TODO(measure): <what needs measuring> instead.

Corollary: never write a findings document before the code that produces it has run. Do not scaffold a report with expected values to be filled in later. That is how invented numbers survive into a final draft.

Evidence taxonomy

Every quantitative claim carries exactly one label:

Label Meaning
MEASURED Computed by our pipeline from our frozen corpus
DERIVED Mathematically derived from MEASURED or declared inputs
SOURCE_FACT Official external fact (a policy date, a model-version date, a licence term)
LITERATURE A number from an external study — not ours until we reproduce it
ASSUMED An explicit business assumption; must live in config.py
SIMULATED Generated enterprise/process behaviour
COUNTERFACTUAL Replay under an alternative policy

The headline rule: no ASSUMED or SIMULATED quantity may support a headline result, empirical finding, benchmark claim or ROI claim. Anywhere else it must be visibly labelled. tools/check_claims.py enforces this mechanically — it is not advisory.

The measurements

ID Measurement Phase
M0 Temporal reconstructability audit — per source 0
M1 Evidence maturity curve 0
M2 Source-completeness shift across 2024-12-10 and 2026-04-15 0
M3A EPSS version-boundary decision discontinuity 0
M3B Early-warning utility vs later-KEV positives (positive-unlabeled) 1
M4 Action-set frontier at matched effort 1
M5 Change-failure rate on real upgrades (sandboxed) 2
M6 Verification sufficiency 2
M7 Agent topology value 6

M0 runs before M1. If a source turns out not to be historically reconstructable, that changes the plan and you must stop and report it, not work around it silently.

Two things that are never called what they are not

  • KEV is a positive-unlabeled proxy, not exploitation ground truth. KEV=1 is strong evidence of exploitation. KEV=0 proves nothing. Never compute a false-negative rate against KEV without stating this bound in the same paragraph.
  • We cannot compute a consumer-side calibration or reliability curve for EPSS. EPSS targets ~2.5–3% observed-exploitation-activity per 30 days; KEV covers ~0.5% of CVEs cumulatively. Scoring one against the other makes EPSS look ~5× overconfident by arithmetic. Use early-warning utility (M3B), never the word calibration.

Architecture discipline

No component enters the system without a measurement that earns it. Before adding a queue, a workflow engine, a cache, a second agent, a database, or a service, write one sentence naming the measurement that forces it. If you cannot, do not add it.

Specifically banned unless a measurement demands them: Kubernetes, Kafka, multi-region, graph databases, agent frameworks adopted for their own sake.

Two boundaries that are architectural, not stylistic:

  • The model proposes; deterministic policy authorises. Whatever decides "may this action execute" is ordinary code with a decision record. Never an LLM.
  • Security boundaries are non-collapsible; agent boundaries are collapsible. Credentials live in tool adapters, never in agents. A single reasoning agent with a read-only evidence adapter and an isolated repo-write adapter has the same isolation as three agents. Never justify a second agent on credential grounds.

SECOND PRIME DIRECTIVE — the engineering gate

A phase is not complete when its measurement runs. It is complete when its code would survive being paged at 3am.

Every phase has four gates, not one: measurement, engineering, operability, adversarial. All four are in docs/ENGINEERING.md Part II. A phase may not be tagged until all four pass. Producing a valid number through code that fails the engineering gate is the same class of defect as producing an invented number: it makes the project look finished when it is not.

The non-negotiable subset, enforced by tools/check_engineering.py:

  • Every declared dependency is imported by shipping code. No decorative dependencies.
  • Config is imported. exec(compile(...)) is banned.
  • Every threshold a decision depends on lives in config.py — once, not twice.
  • Every network, disk and subprocess call has timeouts, bounded retry with backoff, rate-limit handling, corruption detection and resume.
  • Every dropped record is counted and reported. Silent skips are fabricated numbers under a different name.
  • Every results/*.json records run ID, git SHA, corpus manifest digest and config used.
  • mypy --strict clean on src/ and tools/. Coverage floor 90% on src/.
  • Failure-injection tests for every I/O path. No network in tests.

This does not license gold-plating. Architecture discipline below still binds: the code that exists must be excellent; more code must still be earned by a measurement.

Phases 0 and 1 were tagged under the measurement gate alone. Their outstanding engineering debt is itemised in docs/ENGINEERING.md Part IV (R1–R11) and must close before the phase-2 tag.

Safety boundary — non-negotiable

  • No exploit code. No proof-of-concept development. No weaponisation research.
  • No scanning of systems not owned by the operator.
  • The system reasons about response to public advisories. It does not research offence.
  • Phase 2 executes untrusted third-party code (test suites, package lifecycle hooks). It runs only inside the sandbox defined in docs/SANDBOX.md: no host socket, no credentials, ephemeral FS, CPU/memory/wall limits, default-deny network with explicit registry egress. If the sandbox is not ready, the experiment does not run.

Repo layout

pyproject.toml         Dependencies. uv.lock is committed alongside it.
config.py              ALL assumptions. The only place an ASSUMED number may be defined.
corpus/manifest.json   Per-source retrieval URL, timestamp, schema version, digest, licence
scripts/fetch/         One fetcher per source. Idempotent. Digest-verified.
scripts/measure/       One script per measurement. Writes results/<id>.json. Never prose.
results/               Machine-generated JSON. The only legitimate source of a number.
docs/findings/         Prose reports. Every number here must trace to results/ or a label.
src/                   The system itself. Built in later phases.
tools/                 check_claims.py and other gates.
data/                  Raw downloads. GITIGNORED. Never committed.

Never commit raw data. Commit the manifest and the digests so the corpus is reconstructable. data/ is in .gitignore and must stay there.

Commands

make setup      uv sync - resolves and locks the toolchain
make fetch      Download all sources, verify digests, write corpus/manifest.json
make m0         Run the reconstructability audit
make m1 m2 m3a  Run individual measurements
make status     Which phase is next, and what blocks it (reads git tags)
make build      Run every remaining phase headlessly (prompts/AUTONOMOUS_BUILD.md)
make verify     lint + format + typecheck + tests(+coverage) + claims + engineering  <-- the gate
make test       Tests only
make lint       ruff check + ruff format --check
make typecheck  mypy --strict on src/ and tools/

This project uses uv. Run everything through uv run rather than activating a venv, and add dependencies with uv add <pkg> (or uv add --dev <pkg>) rather than editing pyproject.toml by hand. uv.lock is committed - the same reconstructability discipline we apply to the corpus applies to the toolchain.

make verify must pass before you end any turn that touched docs/findings/ or results/. A Stop hook enforces this.

Working rules

  1. Work in phase order. Phases are in docs/DECISIONS.md Part VI (9 phases, 0–8). Do not start a phase whose predecessor's four gates have not all passed (docs/ENGINEERING.md Part II).
  2. Stop and report — do not work around — when:
    • M0 finds a source is not historically reconstructable
    • a kill criterion fires (they are in docs/DECISION.md)
    • a measurement contradicts the plan
    • you need a credential, a paid API, or network access to something not in the manifest
    • a fix would require inventing a number
  3. Commit per measurement, with the results JSON in the same commit as the code that produced it. Message format: M2: measure CPE provenance by publication month.
  4. Write docs/findings/<id>.md only after results/<id>.json exists. Report what the data says, including when it says the hypothesis was wrong. A negative result is a publishable finding, not a failure.
  5. Update docs/findings/STATUS.md after each measurement: what ran, what it showed, what it changed, what it killed.