Skip to content

feat(keeper): retry transient RPC failures on the read path #6

Description

@0dillon

Summary

The public testnet RPC intermittently fails calls for accounts and contracts that exist. Measured
during release verification: getAccount returned Account not found three times in a row for a
funded, live account and then succeeded, all within eight seconds, and a scan on the registry
returned fetch failed and succeeded on an immediate retry. One lkeep extend --footprint run died
on it outright.

The daemon survives this — it logs, moves to the next contract, and picks the work up next tick — so
this is not a correctness bug. But a one-shot lkeep scan or lkeep extend exits non-zero on a
fault that clears in under a second, and a keeper skips a tick's worth of maintenance for the same
reason.

Acceptance Criteria

  • Read-path calls — getLedgerEntries, getAccount, simulateTransaction, registry paging —
    retry a small, documented number of times with backoff.
  • Submission is not retried. sendTransaction and pollTransaction are left alone: a
    resubmission risks paying twice, and extendViaContract already refuses to poll a hash the
    network never queued.
  • Retries are logged at warn with the attempt number, so a flapping endpoint is visible rather
    than hidden.
  • Total time spent retrying is bounded, and cannot make one contract exceed the scan interval.
  • An abort during a retry backoff stops promptly — ctx.signal is already threaded through
    runTick and must be honoured here too.
  • Tests use a fake that fails N times then succeeds; assert the call count, not just the result.

Tech Stack

TypeScript 5.7 on Node 22+ (CI runs 22 and 24), ESM with moduleResolution: "NodeNext" — relative
imports need the .js extension. @stellar/stellar-sdk is pinned to an exact version; verify RPC
API shape against the installed copy, not against documentation. Tests are Vitest under test/ and
run without a network. npm run format:check, npm run lint, npm run build and npm test must
all be clean.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: keepersrc/keeper, the daemon and its decisionscomplexity: mediumHalf a day, some design judgement neededtype: featureAdds behaviour that does not exist yet

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions