Skip to content

LIMITATIONS.md

Latest commit

 

History

History
76 lines (63 loc) · 4.41 KB

LIMITATIONS.md

File metadata and controls

76 lines (63 loc) · 4.41 KB
Raw

Limitations and rejected complexity

This file is the inverse of the README. The README says what the evidence supports; this says where it stops. Both are maintained, because a portfolio that documents only its strengths is documenting its judgement too.

What the hermetic tier proves (339 tests, no services)

  • Deterministic contracts and state transitions.
  • Fail-closed behaviour is executable for policy, budgets, tools, approvals, memory signatures, tenant stores, caches, effects, and sandbox availability.
  • The ARIA reference app composes the same core modules used by the labs.

It proves none of these controls work against the real component. That is the integration tier's job, and the split is deliberate: a green hermetic run is a statement about logic, never about infrastructure.

What the integration tier proves (616 tests, real substrate)

These were previously listed below as unproven. They are now backed by tests that fail without the real component:

  • PostgreSQL transaction and RLS behaviour under a non-owner role. The suite connects as aria_app (NOSUPERUSER NOBYPASSRLS) via DATA_PLANE_DB_URL and asserts cross-tenant denial against FORCE ROW LEVEL SECURITY.
  • Redis admission, leases, and budget reservations against a real server.
  • pgvector storage and recall, including a real embedding model in the live lane.
  • gVisor kernel isolation. 9 @pytest.mark.gvisor tests run against a live rootless runsc Sentry: resource limits, filesystem confinement, egress denial, and the timeout-vs-kill discrimination ladder. Where runsc is absent the provider refuses execution and the tests skip with a reason — unavailable isolation is never a pass.
  • Cedar policy decisions through cedarpy — the real engine, not a re-implementation.
  • MinIO object lock on the WORM audit path.
  • LiteLLM and real provider behaviour in the live lane, against GLM-5.2 (Z.AI) or NVIDIA NIM, whichever credential the environment holds.

What is still not proven

  • Hosted availability, throughput, regional recovery, or SLOs. A compose file is a substrate, not a topology. No load test, no failover drill, no restore drill.
  • Cloud Object Lock retention. MinIO's object lock is exercised; S3/GCS retention semantics under a real compliance hold are not. The local WORM emulator is labelled an emulator everywhere it appears.
  • OTel/Langfuse ingestion. The exporter adapters sit behind the observability seam and are tested against an in-memory exporter. No collector runs in the substrate, so nothing here proves a span survives the wire to a real backend.
  • Model quality. The evaluation plane proves the gate is repeatable — a versioned manifest, a calibrated judge, and a snapshot naming the model that actually scored. It does not establish that any particular model is good.
  • Embeddings on the default provider. The Z.AI coding-plan credential serves chat only; its embedding models answer 1211 Unknown Model. test_live_memory_governance skips for that reason, stated as a capability gap rather than a missing key.
  • Single schema ownership. 18 modules still issue runtime CREATE TABLE alongside the migration job. It works, and it is the wrong shape — the job should be the sole owner. Recorded as accepted debt with a remediation in ADR-0056, not resolved.

Complexity intentionally rejected

  • GraphRAG, without a scenario and a retrieval benchmark to justify it.
  • A semantic cache, without a proven authority-safe similarity boundary.
  • More agents as the default response to low quality. experiments/topology.py exists to make that an experiment; a single agent winning under the constraint is a successful result, not a disappointment.
  • A microVM provider, before gVisor's limitations demand one.
  • LoRA as the privileged improvement path. Prompt, retrieval, routing, and memory proposals all pass through the same evaluation gate.
  • Pretending an in-memory adapter or a local WORM emulator carries a hosted durability or compliance guarantee.

Original ARIA baseline

The source audit found 630 files, 337 Python files, about 820 test functions, and one monolithic commit. 195 contract tests passed in 8.25s. The non-live integration selection did not complete in a five-minute bounded run because it mixed infrastructure-sensitive paths; it is recorded as incomplete, not green.