This system deliberately downloads files from the public internet and will eventually process financially consequential enterprise documents. Two premises follow:

1. Every downloaded byte is hostile until proven otherwise.
2. A false negative is more dangerous than a false positive. Missing a fraudulent invoice costs more than flagging a legitimate one.

Implemented in acquisition/content.py and domain/files.py.

Content that trips a limit moves to QUARANTINED, which is a terminal state. Release requires an explicit human decision, so a bad payload cannot loop back into the pipeline automatically.

Honest about what is not yet built. These are live exposures, not future ones. This section once read "before any crawler is enabled"; the crawler now exists, 8 of 14 sources are approved and enabled, and real documents have been fetched from public portals.

• Decompression bombs. ZIP archives are recognised but not yet expanded. Bounded expansion (entry count, total uncompressed size, nesting depth) must land before any source that serves archives is enabled.
• Malicious PDFs. No PDF sanitisation yet. Parsers must run without network access and with resource limits; embedded JavaScript and external references must be ignored.
• SSRF. Not built. Implemented in acquisition/fetch/: a type-level gate where the transport accepts only a ValidatedTarget, host allowlisting per source, every resolved address validated, mixed answers rejected wholesale, and the validated address pinned to the connection so no second DNS lookup can occur. See the threat model, ADR 0010, and ADR 0011. The transport enforces the address/identity split over real TLS, and is now driven: the crawl runner uses it and 8 of 14 registered sources are approved and enabled, so outbound requests do happen.
• PII detection and redaction. Sources that publish personal data are flagged (contains_personal_data), but no screening is implemented. This must exist before the corpus is used for training.

• Never hardcoded. Configuration comes from the environment via Settings.
• Typed as SecretStr, so they do not leak into repr, logs, or tracebacks.
• safe_database_url() masks passwords for log lines.
• .env is gitignored; .env.example holds only placeholders.
• Production uses a secret manager (AWS Secrets Manager), not files.
• CI runs gitleaks over full history and pip-audit for known vulnerabilities.

docker-compose.yml and .env.example ship well-known passwords. With AEDIFEX_ENVIRONMENT=production, the configuration layer rejects placeholder DSN credentials, the development bucket name, placeholder storage keys, debug=true, and a placeholder User-Agent contact. Failing to start is strictly better than serving with postgres:postgres.

An unrecognised AEDIFEX_* environment variable is also a hard error. A typo such as AEDIFEX_DATABSE_URL would otherwise silently leave the default in place.

• No raw SQL string interpolation. SQLAlchemy parameterises everything.
• Statement timeouts are set at connection level; an unbounded query is how a metadata store takes down an API.
• Container runs as a non-root user. CI asserts this.
• Readiness endpoint leaks nothing. Only the exception type is reported, never the DSN or driver message, since the endpoint is unauthenticated. Tested.
• Inbound X-Request-ID is truncated to 64 characters. It is attacker-controlled and lands in every log line for the request.

As of 2026-08-21 the API has five write endpoints and still has no authentication, no authorization and no tenancy. They exist because the first product workflow needed a beginning: declare a project, upload a document into it, process it, confirm a document's type, record a review. Project identifiers are global UUIDs with nothing scoping them to an owner, so any caller who can reach the port can write into any project.

Two controls stand in the meantime, and neither is a substitute for authorization:

• Writes are refused when the environment is production. require_write_access returns 503, and a test enumerates the application's own POST routes so a write endpoint added later cannot escape the guard. The check is on the environment rather than a feature flag, because a flag is something an operator can switch on without deciding anything — turning this off means deleting the dependency, which means replacing it.
• Uploads are bounded and format-checked before they are stored. The body is read in chunks to a 64 MiB ceiling rather than by an unbounded read(), which would size an allocation from a client-controlled header. The declared format is then checked against the content's leading bytes, and a contradiction is refused — an archive named .pdf reaching the PDF reader is untrusted input arriving at a parser under a false description. A filename never reaches a storage key; those come from the content digest.

What is still missing, and must exist before any deployment beyond a private network: an authenticated identity, a project that belongs to an organisation or tenant, and authorization on every read as well as every write — the read endpoints already expose a corpus whose licence terms differ per source. Tracked for Phase 10. See ADR 0018.

Three authored controls cannot run on a private repository without Advanced Security. They are recorded here as gaps rather than presented as passing checks.

Each is gated so that enabling Advanced Security turns them on via a repository variable rather than a workflow change. Nothing is left as a permanently red check.

The repository will not be made public to obtain free CodeQL. That would publish docs/ip/, including the trade-secret register, and trade-secret protection depends on actually maintaining confidentiality. The IP direction outranks the convenience.

Bandit itself is deliberately not added: it overlaps almost entirely with the bandit-derived S rules already enforced by ruff, and a second tool covering the same ground is maintenance without coverage.

Semgrep runs five rulesets — p/python, p/security-audit, p/secrets, p/dockerfile, p/github-actions — the last two chosen because the mistakes this project has actually made were in a Dockerfile and in workflow permissions, not in application code.

The scan is self-testing. A clean Semgrep run is only meaningful if the engine analysed the code: a wrong path, a failed ruleset fetch, a directory dropped by a default ignore list, or a parse error the tool downgrades to a warning all produce "0 findings" too, and look identical to success. So CI additionally runs .semgrep/selftest.yaml and requires it to produce results — verified by counting matches, files scanned, and scanner errors in the JSON report, not by reading an exit code. Same discipline as REQUIRE_INTEGRATION_TESTS: a check that cannot fail is not a check.

That instrumentation immediately earned its place. Building it surfaced three defects in the gate it was meant to validate, none of which were visible in a green log:

Measured after the fix, in CI: 255 rules over 90 targets, 0 findings, parsed lines ~100.0% (previously ~99.4%, where the missing fraction was the Dockerfile). The self-test reported 119 function definitions across 18 of 28 scanned files in src/aedifex/ with 0 scanner errors, matching the local run exactly. CI enforces floors well below those numbers, so ordinary churn never trips them while a real breakage collapses them to zero.

What the gate does not reach. Test code is outside the scan, because Semgrep's default ignore list excludes test directories. That is acceptable — test code does not ship, and fixtures containing deliberately fake credentials would generate constant noise from p/secrets — but it is a stated limitation rather than an assumed one. Covered: src/, apps/, scripts/, migrations/, the Dockerfile, the workflows, and repository configuration.

Pinning the tool does not pin the rules. semgrep==1.173.0 is pinned, but p/* rulesets are fetched from the registry at run time and change without notice — the rule count moved 255 → 256 between two consecutive local runs. A build can therefore start failing with no change to our code. That is new coverage, and the response is to fix the finding, never to suppress the rule to restore green.

CodeQL is not running. It reports through GitHub code scanning, which on a private repository requires GitHub Advanced Security. This repository does not have it — the API returns Code scanning is not enabled for this repository (HTTP 403), and the first real CI run failed for that reason.

What this means concretely: ruff's bandit-derived rules run on every commit and catch common patterns, but nothing currently follows data flow, so an attacker-controlled value reaching a dangerous sink would not be detected automatically. That is the bug class that matters most for a system built to ingest hostile documents, so this gap is real rather than cosmetic.

The workflow is kept but triggered manually only. A check that can only fail is worse than an acknowledged gap, because it trains people to ignore red.

Three ways to close it, none chosen unilaterally:

Container and filesystem scanning (Trivy) do run, and their SARIF is retained as a build artifact rather than uploaded to code scanning.

Pinning the base image by digest buys reproducibility but freezes its package versions, so security updates must be applied explicitly rather than arriving with a moving tag. That tradeoff is deliberate, and this is the other half of it.

The first real CI Trivy scan found 11 fixable HIGH findings, all in the base image and none in our locked dependencies (every venv package reported zero):

Upstream python:3.13-slim had not been rebuilt — the pinned digest was still the current one — so waiting for a newer digest was not available.

Removing pip is a fix rather than a workaround: the venv is installed non-editable and self-contained, so nothing at runtime needs pip, setuptools, or pkg_resources, and an image that parses hostile documents has no business carrying a package installer. Verified after removal: the package still imports, production hardening still rejects placeholder credentials, alembic still runs, and the API still serves /health and /health/ready.

Public procurement documents may contain PAN, Aadhaar, bank accounts, phone numbers, addresses, signatures, and emails. Obligations:

• Sources known to publish personal data must set contains_personal_data: true.
• Personal data must not be used for model training without screening.
• Licence and permitted use are recorded per source and exposed via the API, so anyone consuming the corpus can see its constraints.

This is a pre-release internal project. Report privately to the maintainers rather than opening a public issue. Do not include real credentials or real project documents in a report.