Skip to content

genesis-v3-resilience-architecture.md

Latest commit

 

History

History
272 lines (193 loc) · 11 KB

genesis-v3-resilience-architecture.md

File metadata and controls

272 lines (193 loc) · 11 KB
Raw

Genesis v3 — Resilience Architecture

Status: Active | Last updated: 2026-03-11

System-level resilience design. Supersedes the tactical patterns in genesis-v3-resilience-patterns.md for architectural decisions; that document remains the building-blocks reference.

1. Scope & Constraints

Genesis runs in a container where only Python + SQLite are guaranteed local. Everything else — LLM providers, Qdrant, Ollama, external APIs — may be cloud or network-adjacent and therefore subject to outage, latency spikes, rate limits, or complete unavailability.

The resilience layer must:

  • Detect partial and total failures across independent axes.
  • Defer work that cannot be completed now without losing it.
  • Queue embedding operations for replay on recovery.
  • Track Claude Code session budgets and throttle proactively.
  • Orchestrate recovery in the correct order when services return.
  • Notify the user through whatever channel remains available.

2. Composite Resilience State Machine

The legacy DegradationLevel enum (L0–L5) collapses multiple failure dimensions into a single ordinal. This makes it impossible to express "cloud LLMs are fine but embeddings are down" without overloading semantics.

2.1 Four Independent Axes

Axis Enum States
Cloud LLM CloudStatus NORMAL → FALLBACK → REDUCED → ESSENTIAL → OFFLINE
Memory MemoryStatus NORMAL → FTS_ONLY → WRITE_QUEUED → DOWN
Embedding EmbeddingStatus NORMAL → QUEUED → UNAVAILABLE
Claude Code CCStatus NORMAL → THROTTLED → RATE_LIMITED → UNAVAILABLE

Each axis is updated independently. The composite ResilienceState is the product of all four axes plus a timestamp and transition history.

2.2 Flapping Protection

If the same axis transitions more than 3 times within a 15-minute window, the state machine holds the worse state for a 10-minute stabilization period. During stabilization, transitions to better states on that axis are suppressed; transitions to worse states are always accepted.

2.3 Legacy Mapping

For backward compatibility, ResilienceState maps to a single DegradationLevel via to_legacy_degradation_level():

  • cloud=NORMAL/FALLBACK → NORMAL/FALLBACK
  • cloud=REDUCED → REDUCED
  • cloud=ESSENTIAL/OFFLINE → ESSENTIAL
  • memory=FTS_ONLY/WRITE_QUEUED/DOWN → MEMORY_IMPAIRED (overrides if worse)
  • embedding=QUEUED/UNAVAILABLE → LOCAL_COMPUTE_DOWN (overrides if worse)

Implementation: src/genesis/resilience/state.py

3. Failure Mode Responses

Failure Mode Detection Response
Single provider failure Circuit breaker opens Route to next in chain; no user impact
Multiple provider failure ≥2 breakers open cloud=REDUCED; defer non-essential work
All cloud down All providers open cloud=OFFLINE; defer all LLM work; notify user
CC rate limits 429 / budget exhaustion cc=THROTTLED→RATE_LIMITED; defer background sessions
Full internet outage All external calls fail All axes degrade; local-only operation
SLM/embeddings down, heavy LLMs fine Ollama unreachable, cloud OK embedding=UNAVAILABLE; queue embeddings; awareness ticks skip SLM
Full chain exhaustion Dead letter overflow cloud=ESSENTIAL; critical-only operation

4. Deferred Work Queue

A SQLite table (deferred_work_queue) stores Genesis-level work items that cannot be completed due to degraded state.

4.1 Schema

See src/genesis/db/schema/_tables.py for DDL. Key fields:

  • work_type: categorizes the work (e.g., reflection, outreach_draft, morning_report)
  • priority: integer 10–100, lower = higher priority
  • staleness_policy: one of drain, refresh, discard, ttl
  • staleness_ttl_s: seconds before TTL-policy items expire

4.2 Staleness Policies

Policy Behavior
drain Always process on recovery, regardless of age
refresh Discard stale item; re-derive from current state on recovery
discard Drop if not processed within a reasonable window
ttl Drop if staleness_ttl_s has elapsed since deferred_at

4.3 Priority Constants

Name Value Use Case
FOREGROUND 10 User-initiated work
URGENT_OUTREACH 20 Blocker/alert outreach
REFLECTION 30 Reflection sessions
SCHEDULED 40 Scheduled tasks
MEMORY_OPS 50 Memory maintenance
OUTREACH_DRAFT 60 Non-urgent outreach
MORNING_REPORT 70 Morning report generation
SURPLUS 80 Surplus/brainstorm work

Implementation: src/genesis/resilience/deferred_work.py

5. Embedding Backlog

When embedding providers (Ollama, Mistral) are unavailable, content that needs embedding is written to pending_embeddings rather than dropped.

5.1 Schema

See src/genesis/db/schema/_tables.py. Each row captures the raw content, memory type, target collection, and tags needed to embed and upsert on recovery.

5.2 Recovery Worker

EmbeddingRecoveryWorker drains the pending queue at a configurable pace (default 10/min) to avoid overwhelming a freshly-recovered embedding provider. On individual item failure, the worker marks that item failed and continues.

Implementation: src/genesis/resilience/embedding_recovery.py

6. CC Session Budget Management

Claude Code sessions are rate-limited by Anthropic. Genesis must:

  1. Track active sessions, completions, and rate-limit responses.
  2. Prioritize foreground > background reflection > background tasks.
  3. Throttle by deferring low-priority background sessions when approaching limits.
  4. Back off on 429s with exponential delay.

The CCStatus axis reflects current CC availability. When RATE_LIMITED, only foreground sessions proceed; all background work is deferred.

7. Recovery Orchestration

When a degraded axis returns to a better state, recovery follows this sequence:

  1. Health confirmation — Verify the service is genuinely back (not just one successful call). Require 3 consecutive successes before upgrading state.
  2. Staleness expiry — Run expire_stale() on the deferred work queue to drop items whose policies say they're no longer relevant.
  3. Embedding drain — EmbeddingRecoveryWorker.drain_pending() processes queued embeddings at controlled pace.
  4. Deferred work drain — Process remaining deferred work items in priority order.
  5. Dead-letter replay — Replay items from dead_letter table (existing infrastructure in genesis.routing.dead_letter).
  6. Catch-up tick — Fire an out-of-cycle awareness tick to re-assess signals with restored capabilities.

8. Out-of-Band Notification

When Genesis enters a degraded state, it needs to notify the user even if the primary conversation channel is unavailable.

8.1 Local Status File

Write a JSON status file to ~/.genesis/status.json on every state transition. Any local tool can read this. Fields: current state, transition history, deferred work count, pending embeddings count, timestamp.

8.2 Webhook (Optional)

If configured, POST state transitions to a webhook URL. Fire-and-forget with 3 retries. Configuration via GENESIS_STATUS_WEBHOOK_URL env var.

8.3 Dashboard (Phase 8)

The health-mcp server (Phase 8) reads resilience state and exposes it via MCP tools for the Agent Zero dashboard.

9. Edge Cases

9.1 Flapping

Handled by the stabilization mechanism in §2.2. The 10-minute hold prevents rapid state oscillation from triggering repeated recovery cycles.

9.2 Partial Recovery

One axis recovers while others remain degraded. Recovery orchestration runs only for the recovered axis's dependent work. Other deferred work stays queued.

9.3 Huge Queue

If deferred_work_queue exceeds 1000 pending items, begin aggressive staleness expiry: treat refresh and discard policies as immediate-expire. Log a warning. This prevents unbounded queue growth during extended outages.

9.4 Outreach Idempotency

Deferred outreach items include a content hash. On drain, check outreach_history for a matching hash before sending. Skip duplicates.

9.5 Stale Morning Report

Morning reports have refresh staleness policy. If the outage spans the morning window, the deferred report is discarded and a fresh one is generated from current state on recovery.

9.6 SQLite Disk Full

If SQLite writes fail with SQLITE_FULL, Genesis enters a hard-degraded state. Log the error, attempt to clear WAL checkpoint, and notify via stderr. No deferred work can be queued — this is a last-resort scenario.

10. Implementation Mapping

10.1 New Files

File Purpose
src/genesis/resilience/__init__.py Package init, public re-exports
src/genesis/resilience/state.py Composite state machine
src/genesis/resilience/deferred_work.py DeferredWorkQueue class
src/genesis/resilience/embedding_recovery.py EmbeddingRecoveryWorker
src/genesis/db/crud/deferred_work.py Deferred work CRUD
src/genesis/db/crud/pending_embeddings.py Pending embeddings CRUD

10.2 Modified Files

File Change
src/genesis/db/schema/_tables.py Add deferred_work_queue and pending_embeddings tables + indexes
src/genesis/observability/types.py Add RESILIENCE to Subsystem enum (if needed)

10.3 Existing Infrastructure

File Relationship
src/genesis/routing/types.py DegradationLevel — legacy mapping target
src/genesis/routing/dead_letter.py Dead letter queue — replayed during recovery
src/genesis/memory/embeddings.py EmbeddingProvider — used by recovery worker
src/genesis/memory/linker.py MemoryLinker — optional auto-linking on embed recovery
src/genesis/qdrant/collections.py upsert_point() — vector storage target
src/genesis/observability/events.py GenesisEventBus — emits resilience events

11. Relationship to health-mcp

The resilience logic lives entirely in src/genesis/resilience/. The health-mcp server (built in Phase 8) is a consumer of resilience state:

  • Reads ResilienceStateMachine.current for dashboard display.
  • Exposes get_resilience_state and get_deferred_work_stats as MCP tools.
  • Does NOT modify resilience state — that's the responsibility of the routing layer and awareness loop.

This separation ensures resilience logic is testable without MCP infrastructure and that health-mcp remains a thin read-only view.

Cross-reference: genesis-v3-resilience-patterns.md for tactical patterns (circuit breakers, retry policies, fallback chains) that this architecture builds upon.

Related Documents