Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions common/slo/__tests__/slo_service_repair.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,70 @@ describe('SloService.repair', () => {
expect(health.invalidateCalls).not.toHaveBeenCalled();
});

it('invalidates the SLO status cache after a successful repair', async () => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p2: the dedup repair invalidation is untested

The new regression test drives getStatus -> repair -> getStatus with setDedupEnabled(false), so it exercises only the single-group repair() invalidate. The identical fix in repairDedup has no test asserting status re-aggregation, and every test in this file runs with dedup disabled. Given the repo's new-line coverage bar, worth adding a setDedupEnabled(true) variant that warms the cache with rules_missing, repairs, and asserts the next read returns ok?

// Regression for the rule_health.spec.js Cypress flake: after Restore,
// the danger callout stayed visible because `liveStatus.state =
// "rules_missing"` remained cached for 60s while the fresh rule-health
// probe already returned `ok`. Repair must drop the cached liveStatus
// alongside the rule-health cache so the next read re-aggregates.
const { ruler, store, deploy } = makeDeps();
const svc = new SloService(noopLogger(), store);
svc.setDedupEnabled(false);

let aggregateCalls = 0;
let aggregateState: 'rules_missing' | 'ok' = 'rules_missing';
svc.setStatusAggregator({
aggregate: async (docs) => {
aggregateCalls += 1;
return docs.map((d) => ({
sloId: d.id,
objectives: [],
state: aggregateState,
firingCount: 0,
ruleCount: 0,
computedAt: new Date().toISOString(),
}));
},
});

const statusCtx = {
client: ({} as unknown) as AlertingOSClient,
workspaceId: 'default',
resolveDatasource: async () => undefined as Datasource | undefined,
};

const doc = await svc.create({ spec: validSpec() }, 'alice', deploy);
ruler.upsertRuleGroup.mockClear();

// Warm the status cache with the broken state — this matches what the
// listing aggregator writes when the user lands on the detail page.
const first = await svc.getStatus(doc.id, statusCtx);
expect(first.state).toBe('rules_missing');
expect(aggregateCalls).toBe(1);

// Cache hit — same answer, no second aggregate call.
const cached = await svc.getStatus(doc.id, statusCtx);
expect(cached.state).toBe('rules_missing');
expect(aggregateCalls).toBe(1);

// Repair flips the ruler state. The next read of liveStatus must pick up
// the new world, not the cached `rules_missing`.
aggregateState = 'ok';
const expected = [
doc.status.provisioning.backend === 'prometheus' && doc.status.provisioning.alertGroupName
? doc.status.provisioning.alertGroupName
: '',
];
const health = makeHealthProbe([missingReport(expected), okReport(expected)]);
const result = await svc.repair(doc.id, { health, deploy });
expect(result.repaired).toBe(true);
expect(result.health.state).toBe('ok');

const after = await svc.getStatus(doc.id, statusCtx);
expect(after.state).toBe('ok');
expect(aggregateCalls).toBe(2);
});

it('first repair flips a missing group to healthy; second repair is a no-op', async () => {
const { ruler, store, deploy } = makeDeps();
const svc = new SloService(noopLogger(), store);
Expand Down
6 changes: 6 additions & 0 deletions common/slo/slo_lifecycle_service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -909,6 +909,11 @@ export class SloLifecycleService {
);

ctx.health.invalidate(ctx.deploy.workspaceId, ctx.deploy.datasource.id, doc.id);
// Status cache holds an aggregated `liveStatus.state` (e.g. 'rules_missing')
// that the detail page falls back to when the fresh rule-health probe is
// 'ok'. Without this, the danger callout re-renders off the stale snapshot
// for up to 60s after a successful repair.
this.statusService.invalidate(doc.id);

const post = await ctx.health.check({
workspaceId: ctx.deploy.workspaceId,
Expand Down Expand Up @@ -1023,6 +1028,7 @@ export class SloLifecycleService {
);

ctx.health.invalidate(ctx.deploy.workspaceId, ctx.deploy.datasource.id, doc.id);
this.statusService.invalidate(doc.id);

const post = await ctx.health.check({
workspaceId: ctx.deploy.workspaceId,
Expand Down
Loading