Skip to content

config-storage.md

Latest commit

 

History

History
514 lines (395 loc) · 19.2 KB

config-storage.md

File metadata and controls

514 lines (395 loc) · 19.2 KB
Raw

Config Storage with Redis

This document describes how to use the Redis-based configuration storage system for the OIF Solver.

Overview

The solver uses Redis as the single source of truth for runtime configuration. Configuration is:

  • Seeded once when deploying a new solver
  • Loaded from Redis on subsequent startups
  • Versioned with optimistic locking for safe updates

Redis Order Indexing And Cleanup

Recovery and transaction bumping load active orders through the canonical is_terminal=false order index. They intentionally do not apply a result limit: missing an active order would be a correctness failure, so the index itself must remain the bounded lookup surface.

The Redis backend rejects NotEquals and NotIn filters instead of resolving them with namespace-wide negative scans. Use positive indexes such as is_terminal=false for liveness paths and add a dedicated indexed query for new production workflows that need Redis-backed filtering.

Redis cleanup runs through the existing solver-engine storage cleanup task using storage.cleanup_interval_seconds. Cleanup walks known Redis sets with SSCAN and prunes stale members whose data key no longer exists. Statically known order sets are scanned directly:

  • <prefix>:orders:_all
  • <prefix>:orders:_index:status_kind:<value>
  • <prefix>:orders:_index:is_terminal:<true|false>
  • <prefix>:orders:<id>:_idx_meta

The Redis backend also records every namespace that writes indexes in <prefix>:_namespaces. Cleanup scans each registered namespace's _all set, so dynamic namespaces such as per-solver bridge transfers keep the same stale-index pruning path after TTL-backed data keys expire.

The final prune rechecks data-key absence inside the Redis script that removes index memberships. If an order is recreated while cleanup is running, cleanup leaves the fresh record's indexes intact.

In Redis cluster mode, the prefix is hash-tagged by the backend, for example {solver-id}:orders:_all, so cleanup operations route to the expected slot.

Terminal order retention is deliberately separate from blanket data TTL. Avoid setting short ttl_orders values for normal operation: active order data must not expire before the maximum settlement and recovery window. Shared Redis index sets and per-key index metadata do not inherit data-key TTL, so positive liveness queries keep their index surface while data keys are live and cleanup can prune stale memberships after data expiry. If terminal history needs bounded retention, prefer an index-safe deletion sweeper that only deletes records after the operator retention period and then removes their index members.

When upgrading Redis data written before the is_terminal order index existed, drain or reindex in-flight orders before relying on is_terminal=false liveness queries. Orders written or updated by current versions self-index on every store.

Useful Redis probes:

redis-cli SCARD '<prefix>:orders:_all'
redis-cli SCARD '<prefix>:orders:_index:is_terminal:false'
redis-cli SCARD '<prefix>:orders:_index:is_terminal:true'

Quick Start

Prerequisites

  • Redis running (default: localhost:6379)
  • Environment variables set:
export REDIS_URL=redis://localhost:6379
export SOLVER_PRIVATE_KEY=your_64_hex_character_private_key

First Run: Seed Configuration

# Seedless configuration (all values from JSON)
cargo run -- --bootstrap-config config/example.json

# Seed testnet configuration (preset fallback for known chains)
cargo run -- --seed testnet --bootstrap-config config/seed-overrides-testnet.json

# Seed mainnet configuration (preset fallback for known chains)
cargo run -- --seed mainnet --bootstrap-config config/seed-overrides-mainnet.json

# Seed using a non-seeded networks JSON example
cargo run -- --seed testnet --bootstrap-config config/non-seeded-networks-example.json

# Or pass JSON directly (useful for deployment services)
cargo run -- --seed testnet --bootstrap-config '{"solver_id":"my-solver","networks":[{"chain_id":11155420,"tokens":[{"symbol":"USDC","address":"0x191688B2Ff5Be8F0A5BCAB3E819C900a810FAaf6","decimals":6}]},{"chain_id":84532,"tokens":[{"symbol":"USDC","address":"0x73c83DAcc74bB8a704717AC09703b959E74b9705","decimals":6}]}]}'

Subsequent Runs: Load from Redis

# Configuration is automatically loaded from Redis
cargo run --

CLI Flags

Flag Description
--seed <preset> Seed configuration using a preset (testnet or mainnet)
--bootstrap-config <value> Bootstrap config as JSON file path OR raw JSON string
--seed-overrides <value> Deprecated alias for --bootstrap-config
--force-seed Overwrite existing configuration in Redis

Bootstrap Config Format

Bootstrap config specifies which networks your solver will support. Networks can be:

  • Preset-backed (mainnet / testnet seed)
  • Non-seeded (new chain IDs) when required fields are provided
{
  "solver_id": "my-solver-instance",
  "monitoring_timeout_seconds": 864000,
  "networks": [
    {
      "chain_id": 11155420,
      "tokens": [
        {
          "symbol": "USDC",
          "address": "0x191688B2Ff5Be8F0A5BCAB3E819C900a810FAaf6",
          "decimals": 6
        }
      ]
    },
    {
      "chain_id": 84532,
      "tokens": [
        {
          "symbol": "USDC",
          "address": "0x73c83DAcc74bB8a704717AC09703b959E74b9705",
          "decimals": 6
        }
      ],
      "rpc_urls": ["https://my-custom-rpc.com"]
    }
  ],
  "admin": {
    "withdrawals": {
      "enabled": true,
      "recipient_allowlist": [
        "0x1111111111111111111111111111111111111111"
      ]
    }
  }
}

Fields

Field Required Description
solver_id No Unique solver identifier. If provided, enables idempotent seeding and derives the EIP-712 admin domain salt that binds admin signatures to this solver instance. If omitted, a UUID is generated.
monitoring_timeout_seconds No Top-level solver monitoring timeout in seconds. Controls how long post-fill settlement monitoring keeps polling for claim readiness. Valid range: 30 to 1209600 seconds. Defaults to seed/common default (28800). Long-latency broadcaster routes may need values like 864000 (10 days).
resource_lock_enabled No Enables ResourceLock quote generation and intake. Defaults to false; leave disabled until ResourceLock reservation semantics are implemented.
networks Yes Array of networks to support
networks[].chain_id Yes Chain ID (seeded or non-seeded)
networks[].tokens Yes Tokens for this network (can be empty at boot)
networks[].tokens[].symbol Yes Token symbol (e.g., "USDC")
networks[].tokens[].address Yes Token contract address
networks[].tokens[].decimals Yes Token decimals
networks[].rpc_urls No Custom RPC URLs (falls back to seed defaults)
admin.withdrawals.enabled No Enables POST /api/v1/admin/withdrawals. Defaults to false.
admin.withdrawals.recipient_allowlist No Pre-approved admin withdrawal recipients. Empty or omitted preserves previous behavior and permits any recipient; non-empty requires each withdrawal recipient to match one listed address exactly.
settlement.type No "hyperlane" (default), "direct", or "broadcaster"
settlement.hyperlane Conditional Required for non-seeded chains when settlement.type = "hyperlane"; include mailboxes, igp_addresses, domains, and oracle maps for every configured chain
settlement.direct Conditional Required when settlement.type = "direct"
settlement.broadcaster Conditional Required when settlement.type = "broadcaster"

OP Stack Extra Native Fee Override

Known OP Stack chains are configured with op_stack_l1_data automatically. For a non-seeded OP Stack-style chain, add an explicit fee-policy override:

{
  "fee_policy": {
    "chains": {
      "747474": {
        "extra_native_fee": {
          "type": "op_stack_l1_data",
          "buffer_bps": 1500
        }
      }
    }
  }
}

oracle_address may also be supplied when the chain does not use the default OP Stack Gas Price Oracle predeploy.

Required Fields For Non-Seeded Networks

For each non-seeded network, provide:

  • name
  • type
  • input_settler_address
  • output_settler_address
  • rpc_urls (at least one URL)

Optional per-network fields:

  • input_settler_compact_address
  • the_compact_address
  • allocator_address

Settlement Examples

hyperlane is the default settlement type when settlement is omitted.

Intent expiry and settlement timing configuration is documented in:

  • docs/oracles/settlement-timing-configuration.md

For broadcaster routes with slow proof availability, tune both:

  • monitoring_timeout_seconds for the post-fill claimability loop
  • settlement timing fields such as settlement.broadcaster.proof_wait_time_seconds and settlement.broadcaster.intent_min_expiry_seconds

Quote expiry is settlement-aware. api.quote.expires_seconds controls the operator-preferred minimum order lifetime, but generated signed orders are clamped upward when the selected settlement implementation requires a longer admission window. Keep api.quote.validity_seconds short for quote retrieval/cache TTL; do not rely on api.quote.expires_seconds to shorten broadcaster orders below settlement.broadcaster.intent_min_expiry_seconds.

For seeded deployments, the bootstrap/seed override model does not currently expose api.quote. Runtime config still receives the default quote settings during operator-config merge, and the admission-safe quote floor is applied from settlement timing regardless.

The key intent_min_expiry_seconds uses the same field name across:

  • settlement.hyperlane.intent_min_expiry_seconds
  • settlement.direct.intent_min_expiry_seconds
  • settlement.broadcaster.intent_min_expiry_seconds

See config/non-seeded-networks-example.json for a full non-seeded Hyperlane example (both chain IDs are non-seeded).

Hyperlane7683 On-Chain Orders

When settlement.type is hyperlane and settlement.hyperlane is configured, the solver enables Hyperlane7683 on-chain discovery for supported EVM and Starknet networks. Use this mode when the orders are opened directly on Hyperlane7683 settler/router contracts instead of submitted through the solver Orders API.

For every supported route:

  • Set each network's input_settler_address to the Hyperlane7683 contract that emits Open.
  • Set each network's output_settler_address to the Hyperlane7683 contract that fills/settles on that chain.
  • Configure settlement.hyperlane.domains with the Hyperlane domain id for each chain. Domain ids are uint32 values and are not always the same as EVM chain ids.
  • Configure settlement.hyperlane.mailboxes and settlement.hyperlane.igp_addresses for each chain that may dispatch settlement messages.
  • Ensure the Hyperlane7683 routers are enrolled for every remote domain and their mailboxes/hooks are configured to quote and accept the settlement dispatch payment.

For Hyperlane7683 EVM claims, the solver prefers a full mailbox fee quote. It reads the destination settler's enrolled router, mailbox, hook, destination gas, and filledOrders(orderId) data, builds the exact settle dispatch body and hook metadata, then calls Mailbox.quoteDispatch(originDomain, router, messageBody, metadata, hook). The returned native-token value is attached to settle([orderId]).

If the full dispatch quote is unavailable, the EVM claim path falls back to the destination settler's quoteGasPayment(originDomain). Pre-order route fee estimates also use quoteGasPayment(originDomain) because the filled order's fillerData does not exist before the solver fills the order. A zero quote is rejected by default because production Hyperlane dispatches generally require payment and underpayment can revert or leave messages undelivered.

Keep this unset or false in production:

{
  "settlement": {
    "type": "hyperlane",
    "hyperlane": {
      "allow_zero_hyperlane7683_settle_quote": false
    }
  }
}

Only set allow_zero_hyperlane7683_settle_quote to true for local mocked tests where the Hyperlane hook intentionally returns a zero quote.

Before a live route smoke test, run the destination-side preflight:

DESTINATION_RPC_URL=https://base-sepolia.example \
DESTINATION_SETTLER=0x... \
ORIGIN_DOMAIN=11155420 \
scripts/flow/preflight-hyperlane7683-route.sh

After a fill, add ORDER_ID, FILLER_DATA, and REFUND_ADDRESS to quote the exact Mailbox.quoteDispatch call that the solver will attach to settle([orderId]).

Example direct settlement:

{
  "networks": [
    { "chain_id": 11155420, "tokens": [] },
    { "chain_id": 84532, "tokens": [] }
  ],
  "settlement": {
    "type": "direct",
    "direct": {
      "dispute_period_seconds": 900,
      "oracle_selection_strategy": "RoundRobin",
      "oracles": {
        "input": {
          "11155420": ["0x7100000000000000000000000000000000000007"],
          "84532": ["0x8200000000000000000000000000000000000008"]
        },
        "output": {
          "11155420": ["0x7100000000000000000000000000000000000007"],
          "84532": ["0x8200000000000000000000000000000000000008"]
        }
      }
    }
  }
}

Note: Providing a solver_id makes seeding idempotent - running bootstrap again with the same config will detect existing configuration and skip seeding (unless --force-seed is used). Keep the solver_id stable after launch; changing it changes the EIP-712 admin domain salt, so existing pending admin signatures will no longer verify.

Environment Variables

Variable Required Default Description
REDIS_URL Yes redis://localhost:6379 Redis connection URL
SOLVER_PRIVATE_KEY Yes - 64-character hex private key (without 0x prefix)
SOLVER_ID For loading - Solver ID to load from Redis (required when not seeding)
JWT_SECRET Conditional - Required when Orders API auth or admin auth is enabled. Must be at least 32 bytes after trimming whitespace

Note: After seeding, the solver outputs the SOLVER_ID to use for subsequent runs. Set this environment variable before running without --bootstrap-config.

Generate production JWT secrets with a high-entropy value, for example:

export JWT_SECRET="$(openssl rand -base64 48)"

When authentication is enabled, the solver refuses to boot if JWT_SECRET is missing or shorter than 32 bytes. This avoids restart-local secrets that invalidate every token and public demo placeholders that could let attackers forge JWTs.

Supported Networks

Testnet Preset

Chain Chain ID Name
Optimism Sepolia 11155420 optimism-sepolia
Base Sepolia 84532 base-sepolia

Mainnet Preset

Chain Chain ID Name
Optimism 10 optimism
Base 8453 base
Arbitrum 42161 arbitrum

You can also seed non-seeded chain IDs with the required non-seeded network fields and settlement config.

How It Works

1. Seeding

When you run with bootstrap flags, the solver:

  1. Optionally loads the seed preset (testnet/mainnet) when --seed is provided
  2. Merges your bootstrap config with defaults
    • seeded chains can reuse seed values
    • non-seeded chains must provide required network bundle and settlement data
  3. Generates a unique solver_id (e.g., solver-abc123-...)
  4. Stores the complete configuration in Redis
┌─────────────────────┐     ┌─────────────────────┐     ┌─────────────────┐
│  Optional Seeds     │     │  Bootstrap Config   │     │  Final Config   │
│  (testnet/mainnet)  │  +  │  (your JSON file)   │  =  │  (in Redis)     │
│                     │     │                     │     │                 │
│  - Contract addrs   │     │  - Chain IDs        │     │  Complete       │
│  - Oracle addrs     │     │  - Tokens           │     │  solver         │
│  - Default RPCs     │     │  - RPC URLs (opt)   │     │  configuration  │
│  - Gas settings     │     │                     │     │                 │
└─────────────────────┘     └─────────────────────┘     └─────────────────┘

2. Loading

On subsequent runs (without bootstrap flags), the solver:

  1. Reads the SOLVER_ID from environment or uses the last seeded ID
  2. Loads the full configuration from Redis
  3. Starts the solver with the loaded configuration

3. Versioning

Configuration in Redis includes version tracking:

{
  "data": { /* full config */ },
  "version": 1,
  "updated_at": 1705849200
}

Updates use optimistic locking - if another process modified the config, your update will fail with a version mismatch error.

Redis Key Structure

{prefix}:config:{solver_id}  →  Versioned<Config>

Default prefix: oif-solver

Example: oif-solver:config:solver-abc123-def456-...

Troubleshooting

"Configuration not found for solver"

The solver ID in your environment doesn't have configuration in Redis. Either:

  • Run with --bootstrap-config to create new configuration
  • Check SOLVER_ID environment variable matches an existing solver

"Configuration already exists"

You're trying to seed when configuration already exists. Use --force-seed to overwrite:

cargo run -- --seed testnet --bootstrap-config config/seed-overrides-testnet.json --force-seed

"Private key must be 64 hex characters"

Ensure your private key:

  • Is exactly 64 hex characters (32 bytes)
  • Does NOT include the 0x prefix
  • Is exported in your shell: export SOLVER_PRIVATE_KEY=...

"Redis connection timeout"

Check that Redis is running and accessible:

redis-cli ping
# Should return: PONG

API Endpoints

When running, the solver exposes these API endpoints:

Endpoint Method Description
/api/v1/tokens GET List supported tokens
/api/v1/quotes POST Request a quote
/api/v1/orders POST Submit an order
/api/v1/orders/{id} GET Get order status

The API server runs on 127.0.0.1:3000 by default.

Integration Tests

Run the integration tests (requires Redis running locally):

cargo test --package solver-storage config_store_integration -- --ignored

Example: Full Setup

# 1. Start Redis
redis-server

# 2. Set environment variables
export REDIS_URL=redis://localhost:6379
export SOLVER_PRIVATE_KEY=your_private_key_here

# 3. Create bootstrap config
cat > config/my-overrides.json << 'EOF'
{
  "networks": [
    {
      "chain_id": 11155420,
      "tokens": [
        {"symbol": "USDC", "address": "0x191688B2Ff5Be8F0A5BCAB3E819C900a810FAaf6", "decimals": 6}
      ]
    },
    {
      "chain_id": 84532,
      "tokens": [
        {"symbol": "USDC", "address": "0x73c83DAcc74bB8a704717AC09703b959E74b9705", "decimals": 6}
      ]
    }
  ]
}
EOF

# 4. Seed configuration
cargo run -- --seed testnet --bootstrap-config config/my-overrides.json

# 5. Subsequent runs just load from Redis
cargo run --