|
| 1 | +## Summary |
| 2 | +WRIT now benchmarks the failure modes that actually break long-lived AI memory in production: source-authority overwrites, extraction drift, system-level write failures, fact lifecycle transitions, and pre-delivery integrity checks. |
| 3 | + |
| 4 | +## Why This Release Exists |
| 5 | +This release expands WRIT in direct response to failure modes surfaced in the r/AIMemory discussion "No AI memory benchmark tests what actually breaks." That discussion highlighted five recurring gaps in real-world memory systems: lower-authority writes overwriting user-stated facts, extraction drift creating near-duplicate records, flush/restart failures corrupting state, facts lacking explicit superseded/expired lifecycle handling, and systems returning stale or conflicting state without certifying integrity before delivery. |
| 6 | + |
| 7 | +These failure classes map directly to the new WRIT dimensions: |
| 8 | +- User correction overwritten by later summaries -> `trust_hierarchy` |
| 9 | +- Same fact re-extracted in slightly different forms -> `extraction_drift` |
| 10 | +- Flush/reset/stale-context corruption -> `failure_injection` |
| 11 | +- Superseded, expired, and reinstated facts -> `lifecycle` |
| 12 | +- Detect stale/conflicting state before answering -> `certification` |
| 13 | + |
| 14 | +## What changed for WRIT users |
| 15 | +- Added 5 new benchmark dimensions: `trust_hierarchy`, `extraction_drift`, `failure_injection`, `lifecycle`, and `certification`. |
| 16 | +- Added 25 new scenarios across those dimensions, expanding the benchmark dataset from 52 to 77 scenarios. |
| 17 | +- Added `closure` coverage to capture resolved-vs-discussed state, including superseded policy and pricing decision scenarios. |
| 18 | +- Expanded aggregate reporting with new metrics for source authority integrity, dedup accuracy, failure resilience, lifecycle accuracy, and pre-delivery detection. |
| 19 | + |
| 20 | +## API surface and contracts |
| 21 | +- Extended `ScenarioCategory`, `RequiredCapability`, `FailureMode`, `ScenarioScores`, and `AggregateScores` in the TypeScript API. |
| 22 | +- Added `source_authority` on memory events. |
| 23 | +- Added `lifecycle_history`, `expected_entity_count`, and `expected_integrity_flag` to scenario ground truth. |
| 24 | +- Extended adapter capabilities with support declarations for source authority, deduplication, lifecycle tracking, and pre-delivery certification. |
| 25 | + |
| 26 | +## Behavior changes |
| 27 | +- The benchmark can now distinguish retrieval failures from write-authority failures, dedup failures, lifecycle blindness, and certification misses. |
| 28 | +- Markdown and JSON reports now surface the new aggregate metrics and category-level breakdowns. |
| 29 | +- Scenario validation now recognizes the new categories and enforces category-specific structural requirements. |
| 30 | + |
| 31 | +## Docs site & CI / tooling |
| 32 | +- Updated WRIT docs for authoring, metrics, and adapter implementation to cover the new dimensions and scenario fields. |
| 33 | +- Fixed a CLI import side effect where `report.ts` executed on module import, which broke the GitHub Actions `benchmark-baseline` workflow when `cli.ts` imported `generateMarkdownReport`. |
| 34 | + |
| 35 | +## Internal changes |
| 36 | +- Refactored evaluator scoring and aggregation to support dimension-specific metrics while preserving null-skipping behavior for unsupported adapter capabilities. |
| 37 | +- Updated built-in adapters to advertise the extended capability surface. |
| 38 | + |
| 39 | +## Fixes |
| 40 | +- Fixed the benchmark CLI / report interaction so `npx tsx src/cli.ts --adapter baseline ...` no longer crashes by treating `--adapter` as a results directory. |
| 41 | + |
| 42 | +## Tests and validation |
| 43 | +- `npx tsc --noEmit` |
| 44 | +- `npx vitest run` |
| 45 | +- Local reproduction of the baseline benchmark CLI run |
| 46 | +- Local verification that standalone `src/report.ts` CLI still works after the entry-point guard fix |
| 47 | + |
| 48 | +## Breaking changes |
| 49 | +- None, but custom adapters and any code that exhaustively matches scenario categories, required capabilities, failure modes, or aggregate score keys must be updated for the expanded WRIT type surface. |
0 commit comments