Skip to content

FIXTURE_DRIFT.md

Latest commit

 

History

History
93 lines (62 loc) · 2.99 KB

FIXTURE_DRIFT.md

File metadata and controls

93 lines (62 loc) · 2.99 KB
Raw

Fixture Drift & Manifest Refresh

Canonical fixtures (VI1.vi, VI2.vi) are tracked by fixtures.manifest.json (bytes + sha256). The validator keeps fixtures deterministic and records auto-refresh events for review.

Validator overview

  • Run locally: pwsh -File tools/Validate-Fixtures.ps1 -Json
  • Key exit codes: 0 ok, 4 size issue, 6 hash mismatch, 5 multiple issues
  • JSON output includes summaryCounts and autoManifest block (written, reason, path)

When both fixtures change, the validator treats it as deterministic drift and writes a new manifest (autoManifest.written=true).

CI usage

The Fixture Drift workflow:

  • Runs strict and override validations (strict.json, override.json)
  • Notes manifest refreshes in the job summary when autoManifest.written is true
  • Uploads the refreshed manifest as an artifact

CI can gate on non-zero exit code or inspect autoManifest.written to flag drift.

Docker runtime manager contract

Fixture Drift stays Linux-only by design. The workflow keeps two Linux jobs:

  • Fixture Drift (Ubuntu) for fixture validator drift checks.
  • NI Linux Container Compare (Hosted) for NI Linux image parity evidence.

Hosted Windows image proof now lives in Validate under the non-required vi-history-scenarios-windows lane so fixture drift stays lightweight while the slower Windows image hydration runs only on the dedicated Windows validation surface.

Linux container evidence is written under:

  • results/fixture-drift/ni-linux-container/**
  • Includes the review suite outputs (history-report.md, history-report.html, history-summary.json, and related inspection artifacts).

Quick local Linux bootstrap command:

node tools/npm/run-script.mjs docker:ni:linux:bootstrap

Manual manifest updates

For intentional fixture updates (outside automation):

pwsh -File tools/Update-FixtureManifest.ps1 -Allow

Include [fixture-update] in the commit message to acknowledge the change.

Optional pair digest block

fixtures.manifest.json can include a deterministic pair block (schema fixture-pair/v1). It captures the combined base/head digest and expected outcome.

Fields:

  • basePath, headPath, algorithm (sha256)
  • canonical, digest
  • expectedOutcome (identical, diff, any)
  • enforce (notice, warn, fail)

Inject locally:

pwsh -File tools/Update-FixtureManifest.ps1 -Allow -InjectPair `
  -SetExpectedOutcome diff `
  -SetEnforce warn

Validate with evidence:

pwsh -File tools/Validate-Fixtures.ps1 -Json -RequirePair -FailOnExpectedMismatch `
  -EvidencePath results/fixture-drift/compare-exec.json

Evidence search order (when -EvidencePath omitted):

  1. results/fixture-drift/compare-exec.json
  2. Latest tests/results/**/(compare-exec.json|lvcompare-capture.json)

Outcome mapping: LVCompare exit code 0 → identical, 1 → diff, else unknown (or use the diff boolean when available).