Show and Tell: Zero-Daemon Crash Recovery for Long-Running DSPy Optimizers (MIPROv2 / Teleprompters)

[sdageltc]

Problem: Costly Crashes in Long Teleprompter Compilations

When optimizing DSPy programs with search-heavy teleprompters (such as BootstrapFewShotWithRandomSearch, MIPROv2, or multi-candidate Bayesian signature search), runs often span hundreds of candidate evaluations over several hours.

If the process crashes at trial 47/50 due to a spot instance interruption, network timeout, or rate-limit error:

• All intermediate candidate evaluations and metric scores are lost.
• Restarting from scratch burns substantial model tokens and API budget re-evaluating identical candidates.

Solution: Zero-Daemon Step-Level Checkpointing (LetItLoop)

We built an open-source, zero-daemon in-process durability library called LetItLoop (GitHub).

By wrapping candidate evaluation steps with @durable and step(), each candidate prompt, program score, and validation result is written to an atomic, CRC32-checksummed Write-Ahead Log (WAL) on disk.

When restarting after a crash:

1. Previously evaluated candidates are fast-forwarded from the WAL in <1ms without touching LLM APIs.
2. The optimizer immediately resumes search from the uncompleted candidates.

Minimal Example: Durable Optimizer Loop

import dspy
from letitloop import durable, step

@durable(goal_id="dspy-mipro-optimization", wal_dir=".bench_wal/dspy")
def optimize_pipeline(trainset, valset, max_candidates=20):
    best_score = -1.0
    best_program = None
    
    for candidate_id in range(1, max_candidates + 1):
        # Step-level checkpointing: skipped automatically on replay if already scored
        result = step(
            f"candidate-{candidate_id}",
            evaluate_candidate_trial,
            candidate_id,
            trainset,
            valset
        )
        if result["score"] > best_score:
            best_score = result["score"]
            best_program = result["program_state"]
            
    return {"best_score": best_score, "program": best_program}

Reference Implementation

We published a complete cookbook with mock teleprompters and automated crash-recovery tests:

• Cookbook: examples/cookbooks/dspy_durable_optimize.py

Curious how others in the DSPy community are persisting long teleprompter search states and handling spot instance restarts during prompt compilations!