Skip to content

Commit 84eb3b1

Browse files
committed
precursor offense tracking
1 parent 3d5edc6 commit 84eb3b1

27 files changed

Lines changed: 2334 additions & 52 deletions

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ Slashmon watches Aztec slashing and has two deliberately separate parts:
66
canonical Aztec contracts and checks slashing rounds directly through public
77
L1 RPCs. Its on-page details panel can select a browser-local RPC and inspect
88
the resolved deployment metadata.
9-
- **PINGME** is the alerting UI for the backend. The backend journals early
10-
offenses from one Aztec node, verifies L1 slashing state, and sends matched
11-
alerts through Telegram or Web Push.
9+
- **PINGME** is the alerting UI for the backend. The backend journals per-duty
10+
inactivity precursors and registered offenses from one Aztec node, verifies
11+
L1 slashing state, and sends matched alerts through Telegram or Web Push.
1212

1313
Node-local offenses are early warnings, not consensus. Slashmon labels them
1414
`pending`. Ethereum observations are labelled `confirmed`. The backend never

collector/.env.example

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ AZTEC_NODE_URL=http://127.0.0.1:8080
88
AZTEC_NODE_API_KEY=
99
AZTEC_ADMIN_URL=http://127.0.0.1:8880
1010
AZTEC_ADMIN_API_KEY=
11+
AZTEC_SENTINEL_POLL_INTERVAL_MS=60000
12+
AZTEC_SENTINEL_LOOKBACK_EPOCHS=3
13+
AZTEC_SENTINEL_VALIDATOR_CONCURRENCY=8
14+
AZTEC_SENTINEL_VALIDATOR_MAX_RESPONSE_BYTES=2097152
1115

1216
# Comma-separated Ethereum RPCs. The network selects the chain and Registry.
1317
L1_RPC_URL=http://127.0.0.1:8545

collector/README.md

Lines changed: 34 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,15 @@
11
# Slashmon backend
22

3-
The backend powers PINGME. It runs as one Node 24 process with four independent
3+
The backend powers PINGME. It runs as one Node 24 process with five independent
44
loops:
55

66
1. poll one Aztec node's public and admin RPCs for pending offenses;
7-
2. scan canonical Ethereum contracts for slashing state and confirmed
7+
2. index each completed epoch's L1 committee and fetch exact-range Sentinel
8+
stats only for those validators;
9+
3. scan canonical Ethereum contracts for slashing state and confirmed
810
`Slashed` logs;
9-
3. deliver the durable outbox through Telegram and Web Push; and
10-
4. long-poll Telegram when that channel is configured.
11+
4. deliver the durable outbox through Telegram and Web Push; and
12+
5. long-poll Telegram when that channel is configured.
1113

1214
Every observation, watch, event, and delivery job lives in one SQLite database.
1315
The browser Monitor does not depend on this process.
@@ -25,7 +27,7 @@ Configuration is intentionally small:
2527
| Area | Variables |
2628
| --- | --- |
2729
| Identity | `SLASHMON_NETWORK`, `SLASHMON_PUBLIC_URL` |
28-
| Aztec node | `AZTEC_NODE_URL`, `AZTEC_NODE_API_KEY`, `AZTEC_ADMIN_URL`, `AZTEC_ADMIN_API_KEY` |
30+
| Aztec node | `AZTEC_NODE_URL`, `AZTEC_NODE_API_KEY`, `AZTEC_ADMIN_URL`, `AZTEC_ADMIN_API_KEY`; optional `AZTEC_SENTINEL_POLL_INTERVAL_MS`, `AZTEC_SENTINEL_LOOKBACK_EPOCHS`, `AZTEC_SENTINEL_VALIDATOR_CONCURRENCY`, `AZTEC_SENTINEL_VALIDATOR_MAX_RESPONSE_BYTES` |
2931
| Ethereum | `L1_RPC_URL`, optional `L1_REGISTRY_ADDRESS`, `L1_SLASH_LOG_LOOKBACK_BLOCKS` |
3032
| Telegram | `TELEGRAM_BOT_TOKEN`, `TELEGRAM_BOT_USERNAME` |
3133
| Web Push | `VAPID_SUBJECT`, `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY` |
@@ -42,6 +44,28 @@ Rollup against its independent L1 view. Missing offenses are acted on only when
4244
the node's relevant L2 cursor advanced safely; positive warnings can still be
4345
recorded during a sync wobble.
4446

47+
The Sentinel indexer is epoch-gated. A normal minute with no newly closable
48+
epoch performs only the node sync check. For each new epoch it resolves
49+
`getEpochCommittee(epoch)` at the exact confirmed L1 block already accepted by
50+
the L1 collector, then calls
51+
`aztec_getValidatorStats(address, epochStart, epochEnd)` for those committee
52+
members with bounded concurrency. On mainnet this is normally 48 targeted node
53+
calls per 32-slot epoch, independent of the roughly 3,200 registered
54+
sequencers.
55+
56+
Each exact-range history is checked against the node's persisted all-time epoch
57+
aggregate. L1 membership supplies explicit `0/0` rows for selected validators
58+
with no recorded status, so zero-duty epochs remain visible and break
59+
inactivity streaks exactly as they do in the node. The first start quietly
60+
indexes the latest three complete epochs by default. If downtime exceeds that
61+
window, both L1 committee reads and Aztec history reads resume at the same
62+
three-epoch boundary and start a new coverage generation, preventing an
63+
unknown gap from extending an inactivity streak.
64+
65+
The authoritative registered offense remains
66+
`aztecAdmin_getSlashOffenses`; its offense type and first-seen time are retained
67+
after it is withdrawn.
68+
4569
Telegram is enabled only when both Telegram variables are present. Web Push is
4670
enabled only when all VAPID variables are present. Generate a stable keypair
4771
with:
@@ -68,7 +92,11 @@ list; deleting a subscription removes the watch and its channels.
6892

6993
`GET /live` reports process liveness. `GET /health` reports operational source
7094
and delivery health. Public events include node-local and L1 observations but never
71-
watch/address associations, provider endpoints, or delivery metadata.
95+
watch/address associations, provider endpoints, or delivery metadata. For an L1
96+
round, `data.nodeEvidence` contains every earlier node offense or completed
97+
inactive epoch matching both the target address and one of the round's target
98+
epochs. This is explicitly correlated node evidence: L1 votes and payloads do
99+
not encode an offense type.
72100

73101
## Storage and delivery
74102

collector/deploy/slashmon-backend.env.example

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ AZTEC_NODE_URL=http://127.0.0.1:8080
99
AZTEC_NODE_API_KEY=
1010
AZTEC_ADMIN_URL=http://127.0.0.1:8880
1111
AZTEC_ADMIN_API_KEY=
12+
AZTEC_SENTINEL_POLL_INTERVAL_MS=60000
13+
AZTEC_SENTINEL_LOOKBACK_EPOCHS=3
14+
AZTEC_SENTINEL_VALIDATOR_CONCURRENCY=8
15+
AZTEC_SENTINEL_VALIDATOR_MAX_RESPONSE_BYTES=2097152
1216

1317
# Comma-separated Ethereum RPCs. The network selects the chain and Registry.
1418
L1_RPC_URL=http://127.0.0.1:8545

collector/src/admin-client.mjs

Lines changed: 205 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
11
import { parseOffenseSnapshot } from './offenses.mjs';
22

3+
const VALIDATOR_STATUSES = new Set([
4+
'checkpoint-mined',
5+
'checkpoint-valid',
6+
'checkpoint-invalid',
7+
'checkpoint-unvalidated',
8+
'checkpoint-missed',
9+
'blocks-missed',
10+
'attestation-sent',
11+
'attestation-missed',
12+
]);
13+
314
export class AztecAdminClient {
415
constructor({
516
url,
@@ -8,6 +19,7 @@ export class AztecAdminClient {
819
nodeApiKey,
920
timeoutMs = 10_000,
1021
maxResponseBytes = 2 * 1024 * 1024,
22+
maxSingleValidatorStatsResponseBytes = 2 * 1024 * 1024,
1123
maxOffenses = 100_000,
1224
fetchImpl = fetch,
1325
}) {
@@ -17,6 +29,7 @@ export class AztecAdminClient {
1729
this.nodeApiKey = nodeApiKey;
1830
this.timeoutMs = timeoutMs;
1931
this.maxResponseBytes = maxResponseBytes;
32+
this.maxSingleValidatorStatsResponseBytes = maxSingleValidatorStatsResponseBytes;
2033
this.maxOffenses = maxOffenses;
2134
this.fetchImpl = fetchImpl;
2235
this.nextId = 1;
@@ -42,10 +55,42 @@ export class AztecAdminClient {
4255
return parseNodeSyncStatus({ ready, l1Timestamp, l2Slot, l2Epoch });
4356
}
4457

58+
async getSentinelSyncStatus(signal) {
59+
const [ready, l2Slot] = await Promise.all([
60+
this.call('aztec_isReady', [], signal),
61+
this.call('aztec_getSyncedL2SlotNumber', [], signal),
62+
]);
63+
const parsed = parseNodeSyncStatus({
64+
ready,
65+
l1Timestamp: null,
66+
l2Slot,
67+
l2Epoch: null,
68+
});
69+
return { ready: parsed.ready, l2Slot: parsed.l2Slot };
70+
}
71+
72+
async getValidatorStats(address, fromSlot, toSlot, signal) {
73+
const sequencer = parseAddress(address, 'validator stats address');
74+
const from = parseUnsignedInteger(fromSlot, 'validator stats fromSlot');
75+
const to = parseUnsignedInteger(toSlot, 'validator stats toSlot');
76+
if (BigInt(from) > BigInt(to)) {
77+
throw new Error('Aztec validator stats fromSlot must not exceed toSlot');
78+
}
79+
const result = await this.call('aztec_getValidatorStats', [sequencer, from, to], signal);
80+
return parseSingleValidatorStats(result, { sequencer, fromSlot: from, toSlot: to });
81+
}
82+
83+
async getInactivityConfig(signal) {
84+
const result = await this.call('aztecAdmin_getConfig', [], signal);
85+
return parseInactivityConfig(result);
86+
}
87+
4588
async call(method, params, signal) {
4689
let request;
4790
if (method === 'aztecAdmin_getSlashOffenses' && isExactParams(params, ['all'])) {
4891
request = { url: this.url, apiKey: this.apiKey, label: 'Aztec admin' };
92+
} else if (method === 'aztecAdmin_getConfig' && isExactParams(params, [])) {
93+
request = { url: this.url, apiKey: this.apiKey, label: 'Aztec admin' };
4994
} else if (
5095
[
5196
'aztec_getNodeInfo',
@@ -56,6 +101,8 @@ export class AztecAdminClient {
56101
].includes(method) && isExactParams(params, [])
57102
) {
58103
request = { url: this.nodeUrl, apiKey: this.nodeApiKey, label: 'Aztec node' };
104+
} else if (method === 'aztec_getValidatorStats' && isValidatorStatsParams(params)) {
105+
request = { url: this.nodeUrl, apiKey: this.nodeApiKey, label: 'Aztec node' };
59106
} else {
60107
throw new Error(`Aztec RPC method or parameters are not allowed: ${method}`);
61108
}
@@ -84,7 +131,10 @@ export class AztecAdminClient {
84131
throw new Error(`${request.label} request failed: ${error instanceof Error ? error.message : String(error)}`);
85132
}
86133

87-
const body = await readLimitedBody(response, this.maxResponseBytes, request.label);
134+
const responseLimit = method === 'aztec_getValidatorStats'
135+
? this.maxSingleValidatorStatsResponseBytes
136+
: this.maxResponseBytes;
137+
const body = await readLimitedBody(response, responseLimit, request.label);
88138
let payload;
89139
try {
90140
payload = JSON.parse(body);
@@ -139,12 +189,136 @@ export function parseNodeSyncStatus(value) {
139189
};
140190
}
141191

192+
export function parseSingleValidatorStats(value, {
193+
sequencer: expectedSequencer,
194+
fromSlot,
195+
toSlot,
196+
} = {}) {
197+
// The node returns undefined (JSON null) when this address has no slot-level
198+
// history. L1 committee membership still lets the collector persist a 0/0
199+
// epoch row after another committee response proves the epoch was evaluated.
200+
if (value === undefined || value === null) return undefined;
201+
if (!isPlainObject(value) || !isPlainObject(value.validator)) {
202+
throw new Error('Aztec single-validator stats must include a validator object');
203+
}
204+
const expected = parseAddress(expectedSequencer, 'expected validator stats address');
205+
const sequencer = parseAddress(value.validator.address, 'validator stats address');
206+
if (sequencer !== expected) {
207+
throw new Error(`Aztec validator stats address does not match requested address ${expected}`);
208+
}
209+
if (!Array.isArray(value.validator.history)) {
210+
throw new Error(`Aztec validator history for ${sequencer} must be an array`);
211+
}
212+
const lowerBound = BigInt(parseUnsignedInteger(fromSlot, 'validator stats fromSlot'));
213+
const upperBound = BigInt(parseUnsignedInteger(toSlot, 'validator stats toSlot'));
214+
const seenSlots = new Set();
215+
let previousSlot;
216+
const history = value.validator.history.map((observation, index) => {
217+
if (!isPlainObject(observation) || !VALIDATOR_STATUSES.has(observation.status)) {
218+
throw new Error(`Aztec validator history status at ${sequencer}[${index}] is invalid`);
219+
}
220+
const slot = parseUnsignedInteger(observation.slot, `validator history slot at ${sequencer}[${index}]`);
221+
const numericSlot = BigInt(slot);
222+
if (numericSlot < lowerBound || numericSlot > upperBound) {
223+
throw new Error(`Aztec validator history slot ${slot} is outside the requested range`);
224+
}
225+
if (seenSlots.has(slot)) {
226+
throw new Error(`Aztec validator history contains duplicate slot ${slot} for ${sequencer}`);
227+
}
228+
if (previousSlot !== undefined && numericSlot <= previousSlot) {
229+
throw new Error(`Aztec validator history is not strictly ordered for ${sequencer}`);
230+
}
231+
seenSlots.add(slot);
232+
previousSlot = numericSlot;
233+
return { slot, status: observation.status };
234+
});
235+
if (!Array.isArray(value.allTimeEpochPerformance)) {
236+
throw new Error(`Aztec all-time epoch performance for ${sequencer} must be an array`);
237+
}
238+
const seenEpochs = new Set();
239+
const allTimeEpochPerformance = value.allTimeEpochPerformance.map((performance, index) => {
240+
if (!isPlainObject(performance)) {
241+
throw new Error(`Aztec epoch performance at ${sequencer}[${index}] is invalid`);
242+
}
243+
const epoch = parseUnsignedInteger(performance.epoch, `validator epoch at ${sequencer}[${index}]`);
244+
if (seenEpochs.has(epoch)) {
245+
throw new Error(`Aztec validator epoch performance contains duplicate epoch ${epoch} for ${sequencer}`);
246+
}
247+
seenEpochs.add(epoch);
248+
const missed = parseSafeInteger(
249+
performance.missed,
250+
`validator missed duties at ${sequencer}[${index}]`,
251+
0,
252+
);
253+
const total = parseSafeInteger(
254+
performance.total,
255+
`validator total duties at ${sequencer}[${index}]`,
256+
0,
257+
);
258+
if (missed > total) {
259+
throw new Error(`Aztec validator missed duties exceed total duties at ${sequencer}[${index}]`);
260+
}
261+
return {
262+
epoch,
263+
missed,
264+
total,
265+
};
266+
});
267+
const totalSlots = parseSafeInteger(value.validator.totalSlots, 'Aztec validator totalSlots', 0);
268+
if (totalSlots !== history.length) {
269+
throw new Error(`Aztec validator totalSlots does not match history length for ${sequencer}`);
270+
}
271+
return {
272+
sequencer,
273+
history,
274+
allTimeEpochPerformance,
275+
lastProcessedSlot: parseOptionalInteger(value.lastProcessedSlot, 'validator stats lastProcessedSlot'),
276+
};
277+
}
278+
279+
export function parseInactivityConfig(value) {
280+
if (!isPlainObject(value)) {
281+
throw new Error('Aztec admin config must be an object');
282+
}
283+
const targetPercentage = value.slashInactivityTargetPercentage;
284+
if (
285+
typeof targetPercentage !== 'number' ||
286+
!Number.isFinite(targetPercentage) ||
287+
targetPercentage < 0 ||
288+
targetPercentage > 1
289+
) {
290+
throw new Error('Aztec admin slashInactivityTargetPercentage must be between 0 and 1');
291+
}
292+
return {
293+
targetPercentage,
294+
consecutiveEpochThreshold: parseSafeInteger(
295+
value.slashInactivityConsecutiveEpochThreshold,
296+
'Aztec admin slashInactivityConsecutiveEpochThreshold',
297+
1,
298+
),
299+
epochEndBufferSlots: parseSafeInteger(
300+
value.sentinelEpochEndBufferSlots,
301+
'Aztec admin sentinelEpochEndBufferSlots',
302+
0,
303+
),
304+
};
305+
}
306+
142307
function isExactParams(actual, expected) {
143308
return Array.isArray(actual)
144309
&& actual.length === expected.length
145310
&& actual.every((value, index) => value === expected[index]);
146311
}
147312

313+
function isValidatorStatsParams(params) {
314+
return Array.isArray(params) &&
315+
params.length === 3 &&
316+
/^0x[0-9a-f]{40}$/.test(params[0]) &&
317+
/^[0-9]+$/.test(params[1]) &&
318+
/^[0-9]+$/.test(params[2]) &&
319+
BigInt(params[1]) <= BigInt(params[2]);
320+
}
321+
148322
function isPlainObject(value) {
149323
return Boolean(value) && typeof value === 'object' && !Array.isArray(value);
150324
}
@@ -175,6 +349,36 @@ function parseIdentityAddress(value, name) {
175349
return value.toLowerCase();
176350
}
177351

352+
function parseAddress(value, label) {
353+
if (typeof value !== 'string' || !/^0x[0-9a-fA-F]{40}$/.test(value) || /^0x0{40}$/i.test(value)) {
354+
throw new Error(`Aztec ${label} must be a nonzero 20-byte hex address`);
355+
}
356+
return value.toLowerCase();
357+
}
358+
359+
function parseOptionalInteger(value, label) {
360+
if (value === undefined || value === null) return undefined;
361+
return parseUnsignedInteger(value, label);
362+
}
363+
364+
function parseUnsignedInteger(value, label) {
365+
if (typeof value === 'number' && Number.isSafeInteger(value) && value >= 0) {
366+
return String(value);
367+
}
368+
if (typeof value === 'string' && (/^[0-9]+$/.test(value) || /^0x[0-9a-f]+$/i.test(value))) {
369+
return BigInt(value).toString();
370+
}
371+
throw new Error(`Aztec ${label} must be an unsigned integer`);
372+
}
373+
374+
function parseSafeInteger(value, label, minimum) {
375+
const parsed = Number(value);
376+
if (!Number.isSafeInteger(parsed) || parsed < minimum) {
377+
throw new Error(`${label} must be an integer of at least ${minimum}`);
378+
}
379+
return parsed;
380+
}
381+
178382
function parseOptionalUnsignedInteger(value, label) {
179383
if (value === null) return undefined;
180384
let parsed;

0 commit comments

Comments
 (0)