fix(slo): invalidate SLO status cache after repair - #2699
Conversation
Repair only invalidated the rule-health probe cache. The 60s SLO status cache held the stale `liveStatus.state = 'rules_missing'` snapshot, and the detail page falls back to it whenever the fresh rule-health probe is 'ok' (slo_detail_page.tsx:691-699). Result: the danger callout re-rendered for up to 60s after a successful Restore. Adds `statusService.invalidate(doc.id)` after the ruler upsert in both `repair` (single-group) and `repairDedup`, matching the pattern already used by every other lifecycle method (create / update / delete / enable / disable). Adds a regression test that drives `getStatus → repair → getStatus` through the public `SloService` API and asserts the second read re-aggregates instead of returning a stale 'rules_missing'. Reproduces the rule_health.spec.js Cypress flake at the unit level: without the fix, the new test fails with "Expected: 'ok' / Received: 'rules_missing'" — same failure shape the spec sees on EuiCallOut--danger continuously found. Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
| expect(health.invalidateCalls).not.toHaveBeenCalled(); | ||
| }); | ||
|
|
||
| it('invalidates the SLO status cache after a successful repair', async () => { |
There was a problem hiding this comment.
p2: the dedup repair invalidation is untested
The new regression test drives getStatus -> repair -> getStatus with setDedupEnabled(false), so it exercises only the single-group repair() invalidate. The identical fix in repairDedup has no test asserting status re-aggregation, and every test in this file runs with dedup disabled. Given the repo's new-line coverage bar, worth adding a setDedupEnabled(true) variant that warms the cache with rules_missing, repairs, and asserts the next read returns ok?
ps48
left a comment
There was a problem hiding this comment.
fix(slo): invalidate status cache after repair
Verified at the head SHA. Repair previously dropped only the rule-health probe cache, leaving the status cache holding a stale rules_missing that the detail page falls back to when the fresh probe reads ok, keeping the danger callout up for the cache TTL. The fix adds statusService.invalidate(doc.id) after the ruler upsert in both repair and repairDedup, matching the pattern already used by create/update/delete/enable/disable. Confirmed the key matches the id-keyed statusCache, and invalidation sits on the success path only, which is correct since ok/ruler_unreachable/throw exits leave ruler state unchanged. Change is confined to common/, no new deps/routes/saved objects.
One coverage gap inline. No p0/p1.
Summary
liveStatus.state = 'rules_missing'snapshot, and the detail page falls back to it whenever the fresh rule-health probe returns'ok'(slo_detail_page.tsx:691-699). Result: the danger callout re-rendered for up to 60s after a successful Restore, causingrule_health.spec.jsto flake on theExpected <div.euiCallOut.euiCallOut--danger> not to existassertion.this.statusService.invalidate(doc.id)after the ruler upsert in bothrepair(single-group) andrepairDedup, matching the pattern already used by every other lifecycle method (create / update / delete / enable / disable).getStatus → repair → getStatusthrough the publicSloServiceAPI and asserts the second read re-aggregates instead of returning a stale'rules_missing'. Without the fix the new test fails withExpected: 'ok' / Received: 'rules_missing'— the same shape the Cypress spec sees.Failure context
Original CI failure: https://github.com/opensearch-project/dashboards-observability/actions/runs/26532215393/job/78151594782
"Continuously found" is the smoking gun — the DOM is in a stable bad state, not slow-loading. The Restore handler's
await load()re-fetches the SLO doc, buthandleGetSLOreturns the cachedliveStatus.state = 'rules_missing'written by the listing aggregator within the prior 60s. The detail-page render atslo_detail_page.tsx:697then re-shows the danger callout from the staleliveStatuseven thoughruleHealth.state === 'ok'.Test plan
yarn test --testPathPattern="slo_service_repair|slo_service\.test|slo_status_writeback|slo_lifecycle"— 48/48 pass with the fix; the new regression test fails without it (Expected: 'ok' / Received: 'rules_missing').yarn test --testPathPattern="server/(routes|services)/slo"— 178/178 pass.yarn test --testPathPattern="public/components/apm/pages/slos/__tests__"— 353/353 pass.cypress-slo-cortex.ymlagainst this branch on the lezzago fork — run 26600851291 —✔ All specs passed!includingrule_health.spec.js ✔ 1/1 passing (03:06).