Skip to content

fix(slo): invalidate SLO status cache after repair - #2699

Open
lezzago wants to merge 1 commit into
opensearch-project:mainfrom
lezzago:cypress-rule-health-fix-validation
Open

fix(slo): invalidate SLO status cache after repair#2699
lezzago wants to merge 1 commit into
opensearch-project:mainfrom
lezzago:cypress-rule-health-fix-validation

Conversation

@lezzago

@lezzago lezzago commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Repair only invalidated the rule-health probe cache. The 60s SLO status cache held the stale liveStatus.state = 'rules_missing' snapshot, and the detail page falls back to it whenever the fresh rule-health probe returns 'ok' (slo_detail_page.tsx:691-699). Result: the danger callout re-rendered for up to 60s after a successful Restore, causing rule_health.spec.js to flake on the Expected <div.euiCallOut.euiCallOut--danger> not to exist assertion.
  • Adds this.statusService.invalidate(doc.id) after the ruler upsert in both repair (single-group) and repairDedup, matching the pattern already used by every other lifecycle method (create / update / delete / enable / disable).
  • Adds a regression test that drives getStatus → repair → getStatus through the public SloService API and asserts the second read re-aggregates instead of returning a stale 'rules_missing'. Without the fix the new test fails with Expected: 'ok' / Received: 'rules_missing' — the same shape the Cypress spec sees.

Failure context

Original CI failure: https://github.com/opensearch-project/dashboards-observability/actions/runs/26532215393/job/78151594782

AssertionError: Timed out retrying after 30000ms:
Expected <div.euiCallOut.euiCallOut--danger> not to exist in the DOM,
but it was continuously found.
  at rule_health.spec.js:230

"Continuously found" is the smoking gun — the DOM is in a stable bad state, not slow-loading. The Restore handler's await load() re-fetches the SLO doc, but handleGetSLO returns the cached liveStatus.state = 'rules_missing' written by the listing aggregator within the prior 60s. The detail-page render at slo_detail_page.tsx:697 then re-shows the danger callout from the stale liveStatus even though ruleHealth.state === 'ok'.

Test plan

  • Unit: yarn test --testPathPattern="slo_service_repair|slo_service\.test|slo_status_writeback|slo_lifecycle" — 48/48 pass with the fix; the new regression test fails without it (Expected: 'ok' / Received: 'rules_missing').
  • Server-side breadth: yarn test --testPathPattern="server/(routes|services)/slo" — 178/178 pass.
  • Front-end breadth: yarn test --testPathPattern="public/components/apm/pages/slos/__tests__" — 353/353 pass.
  • End-to-end: dispatched cypress-slo-cortex.yml against this branch on the lezzago fork — run 26600851291✔ All specs passed! including rule_health.spec.js ✔ 1/1 passing (03:06).

Repair only invalidated the rule-health probe cache. The 60s SLO status
cache held the stale `liveStatus.state = 'rules_missing'` snapshot, and
the detail page falls back to it whenever the fresh rule-health probe is
'ok' (slo_detail_page.tsx:691-699). Result: the danger callout
re-rendered for up to 60s after a successful Restore.

Adds `statusService.invalidate(doc.id)` after the ruler upsert in both
`repair` (single-group) and `repairDedup`, matching the pattern already
used by every other lifecycle method (create / update / delete /
enable / disable).

Adds a regression test that drives `getStatus → repair → getStatus`
through the public `SloService` API and asserts the second read
re-aggregates instead of returning a stale 'rules_missing'.

Reproduces the rule_health.spec.js Cypress flake at the unit level:
without the fix, the new test fails with "Expected: 'ok' / Received:
'rules_missing'" — same failure shape the spec sees on
EuiCallOut--danger continuously found.

Signed-off-by: Ashish Agrawal <ashisagr@amazon.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 No multiple PR themes
⚡ No major issues detected

@github-actions

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Mock datasource instead of undefined

The resolveDatasource function always returns undefined, which may cause the test to
not accurately reflect real-world behavior. Consider returning a mock datasource
object to ensure the test validates the actual code path that will be executed in
production.

common/slo/tests/slo_service_repair.test.ts [382-386]

 const statusCtx = {
   client: ({} as unknown) as AlertingOSClient,
   workspaceId: 'default',
-  resolveDatasource: async () => undefined as Datasource | undefined,
+  resolveDatasource: async () => ({ id: 'test-ds', name: 'test' }) as Datasource,
 };
Suggestion importance[1-10]: 3

__

Why: While providing a mock datasource could be more realistic, the test is specifically focused on cache invalidation behavior after repair. The resolveDatasource returning undefined doesn't appear to break the test's core purpose, and the current implementation may be intentional for this specific test case.

Low

expect(health.invalidateCalls).not.toHaveBeenCalled();
});

it('invalidates the SLO status cache after a successful repair', async () => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p2: the dedup repair invalidation is untested

The new regression test drives getStatus -> repair -> getStatus with setDedupEnabled(false), so it exercises only the single-group repair() invalidate. The identical fix in repairDedup has no test asserting status re-aggregation, and every test in this file runs with dedup disabled. Given the repo's new-line coverage bar, worth adding a setDedupEnabled(true) variant that warms the cache with rules_missing, repairs, and asserts the next read returns ok?

@ps48 ps48 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix(slo): invalidate status cache after repair

Verified at the head SHA. Repair previously dropped only the rule-health probe cache, leaving the status cache holding a stale rules_missing that the detail page falls back to when the fresh probe reads ok, keeping the danger callout up for the cache TTL. The fix adds statusService.invalidate(doc.id) after the ruler upsert in both repair and repairDedup, matching the pattern already used by create/update/delete/enable/disable. Confirmed the key matches the id-keyed statusCache, and invalidation sits on the success path only, which is correct since ok/ruler_unreachable/throw exits leave ruler state unchanged. Change is confined to common/, no new deps/routes/saved objects.

One coverage gap inline. No p0/p1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants