Skip to content

feat(tools): add external dataset replay harness - #56

Merged
jscott3201 merged 2 commits into
mainfrom
tools/external-dataset-harness
Aug 20, 2026
Merged

feat(tools): add external dataset replay harness#56
jscott3201 merged 2 commits into
mainfrom
tools/external-dataset-harness

Conversation

@jscott3201

@jscott3201 jscott3201 commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • add an optional offline external-dataset inspection and replay harness
  • add the first inventory-backed adapter contract for the LBNL simulated PFPU/SFPU dataset
  • add a bounded JSON trace mode to the existing Rust verifier so Python never reimplements CXF/CDL logic
  • add a stable result schema, manual-acquisition and mapping documentation, and future-adapter guidance
  • add only original tiny PFPU/SFPU fixtures; no source archive, inventory PDF, or derived large replay is committed

Architecture and evidence boundaries

Decision Result
Rule execution Temporary gated vectors call the existing pinned open-control engine through cxf-verify --trace-json.
Mapping A required local manifest records case/point/gate inventory evidence, exact columns, units, proxies, dependencies, subtype, labels, fault start, and source version; names are never guessed.
Independence Command/proof, actual/reference, actual/expected, measurement/setpoint, inlet/outlet, and gate/outcome circularity is rejected. A documented topology-equivalent SFPU primary/fan-flow alias is the only narrow exception.
Baselines Expected fan flow and coil delta-T require model readiness/version/input/known-good/error/update metadata.
Timeline Duplicate, out-of-order, and irregular timestamps block replay; gaps and missing required inputs reset both engine state and startup lead. Booleans are never interpolated.
Gating Six required host-gate categories retain fixed minimum applicability; additional independent inventory-backed gates remain supported.
Metrics Healthy rates use fault-free evaluable samples. TPR uses declared inventory rule-case pairs, post-fault evaluability, and post-fault alarm onsets; pre-alarmed windows do not count as detections.
Identity Results record dataset pre/post fingerprint, library revision/dirty state, engine pin and clean build revision, verifier/tool/rule/card/point hashes, and engine-exported content IDs.
Scale/safety Cases run sequentially; trace batches, sample count, runtime, and output size are bounded. Fallback Cargo execution is offline. Writes are restricted away from source/canonical trees.

Source policy

Source Use in this PR Not claimed
Official LBNL simulated FPU dataset, DOI 10.25984/1881324 Adapter target, topology/fault-class inventory contract, future empirical replay No source archive was locally available; no measured FPR/TPR or threshold validation
Existing FPB rule cards/graphs Committed rule semantics, content IDs, engine replay targets No Python reimplementation or threshold change
Original tiny fixtures Contract, failure-mode, and pinned-engine smoke tests Not redistributed LBNL data and not field-performance evidence

Validation

  • dataset harness contract tests: 32 passed
  • Rust trace-boundary tests: 2 passed
  • real pinned-engine tiny fixture replay: six FPB rules across one healthy PFPU and one restricted-flow SFPU
  • fixture result: zero healthy alarm samples; FPB-0004 detected after 900 s; declared/evaluable/detected target-pair counts all pinned
  • result JSON validated against tools/dataset_harness/result.schema.json
  • full verifier: 134 rule packages, all scenarios passed
  • registry lint: 134 rules OK
  • SVG fitment: 135 SVGs clean
  • existing simharness: 6 passed
  • Cargo test/check and Python compile checks
  • mdBook: 134 fault pages, 21 playbooks, 13 point dictionaries
  • workflow YAML, JSON, and git diff --check

External-data posture

No LBNL source archive was present locally. This PR records only original-fixture behavior and makes no full-data FPR, TPR, threshold-tuning, or card-validation claim. Users obtain data manually from the official source and follow its current license and redistribution terms. Full PFPU/SFPU mapping remains contingent on a reviewed local inventory-backed manifest and independently available evidence channels.

@jscott3201
jscott3201 merged commit 41f7d56 into main Aug 20, 2026
3 checks passed
@jscott3201
jscott3201 deleted the tools/external-dataset-harness branch August 22, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant