Skip to content

Commit 0bffa7b

Browse files
authored
fix(retry): word-boundary regex + RetryConfig validation (#64)
Rebased on main (post-#63). Adds RetryConfig.__post_init__ validation for negative field values, uses the named _TRANSIENT_STATUS_RE constant instead of an inline regex, keeps all tests from both sides including the ConnectionError/TimeoutError subclass tests from #63.
1 parent 9ce9599 commit 0bffa7b

4 files changed

Lines changed: 88 additions & 83 deletions

File tree

README.md

Lines changed: 46 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# self-heal
22

3-
[CI](https://github.com/Johin2/self-heal/actions/workflows/ci.yml)
4-
[PyPI](https://pypi.org/project/self-heal-llm/)
5-
[Python](https://pypi.org/project/self-heal-llm/)
6-
[License: MIT](LICENSE)
3+
[![CI](https://github.com/Johin2/self-heal/actions/workflows/ci.yml/badge.svg)](https://github.com/Johin2/self-heal/actions/workflows/ci.yml)
4+
[![PyPI](https://img.shields.io/pypi/v/self-heal-llm.svg)](https://pypi.org/project/self-heal-llm/)
5+
[![Python](https://img.shields.io/pypi/pyversions/self-heal-llm.svg)](https://pypi.org/project/self-heal-llm/)
6+
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
77

88
> Automatic repair for failing Python code, powered by any LLM.
99
10-
self-heal demo
10+
![self-heal demo](assets/demo.gif)
1111

1212
`self-heal` catches failures, proposes an LLM-guided fix with memory of prior attempts, verifies it, and retries. Works with Claude, OpenAI, Gemini, and 100+ other providers. Sync and async. One decorator.
1313

@@ -32,23 +32,19 @@ Two suites, both run against Gemini 2.5 Flash, 3 max attempts, v0.4 harness.
3232

3333
**Default suite** (19 hand-written bugs: price parsing, palindrome, flatten, roman numerals, camelCase-to-snake_case, Levenshtein, anagram, duration formatting, ...)
3434

35-
36-
| Strategy | Tasks passed | Success rate | LLM calls |
37-
| ----------------------------------- | ------------ | ------------ | --------- |
38-
| Naive single-shot repair | 16 / 19 | 84% | 17 |
39-
| **self-heal (multi-turn + memory)** | **18 / 19** | **95%** | 21 |
40-
35+
| Strategy | Tasks passed | Success rate | LLM calls |
36+
|---|---:|---:|---:|
37+
| Naive single-shot repair | 16 / 19 | 84% | 17 |
38+
| **self-heal (multi-turn + memory)** | **18 / 19** | **95%** | 21 |
4139

4240
**QuixBugs** (31 classic one-line bugs; 9 graph/tree programs skipped by the loader for custom-deserialization reasons)
4341

42+
| Strategy | Tasks passed | Success rate | LLM calls |
43+
|---|---:|---:|---:|
44+
| Naive single-shot repair | 27 / 31 | 87% | 30 |
45+
| **self-heal (multi-turn + memory)** | **29 / 31** | **94%** | 35 |
4446

45-
| Strategy | Tasks passed | Success rate | LLM calls |
46-
| ----------------------------------- | ------------ | ------------ | --------- |
47-
| Naive single-shot repair | 27 / 31 | 87% | 30 |
48-
| **self-heal (multi-turn + memory)** | **29 / 31** | **94%** | 35 |
49-
50-
51-
Reproduce: `self-heal bench --proposer gemini --model gemini-2.5-flash` (default) or `--suite quixbugs`. Full numbers, historical rows, and how to contribute your own in `[benchmarks/RESULTS.md](benchmarks/RESULTS.md)`. Task source in `[benchmarks/tasks.py](benchmarks/tasks.py)`.
47+
Reproduce: `self-heal bench --proposer gemini --model gemini-2.5-flash` (default) or `--suite quixbugs`. Full numbers, historical rows, and how to contribute your own in [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md). Task source in [`benchmarks/tasks.py`](benchmarks/tasks.py).
5248

5349
The +2 tasks on each suite share a pattern: the first proposed fix handles one edge case but misses another. Memory of the failed attempt plus test feedback lets the second proposal cover both. Roughly 20% more LLM calls for the additional wins. As frontier models keep improving the naive floor rises and this delta compresses; earlier runs against Gemini 2.5 Flash had naive at 68% instead of 84%, which is honest signal not cherry-picked.
5450

@@ -72,28 +68,24 @@ pip install 'self-heal-llm[all]' # everything
7268
7369
## Provider support
7470

75-
76-
| Adapter | Covers |
77-
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
78-
| `ClaudeProposer` | Anthropic Claude (native SDK) |
79-
| `CohereProposer` | Cohere (native SDK) |
80-
| `OpenAIProposer` | OpenAI + **any OpenAI-compatible endpoint** (OpenRouter, Together, Groq, Fireworks, Anyscale, Perplexity, xAI, DeepSeek, Azure, Ollama, LM Studio, vLLM, llama.cpp server, ...) |
81-
| `GroqProposer` | Groq (OpenAI-compatible; reads `GROQ_API_KEY`, defaults to Llama 3.3 70B) |
82-
| `FireworksProposer` | Fireworks AI (Llama, Qwen, Mixtral, DeepSeek hosted models) |
83-
| `TogetherProposer` | Together AI (Llama, Qwen, DeepSeek hosted models) |
84-
| `GeminiProposer` | Google Gemini (native SDK) |
85-
| `MistralProposer` | Mistral AI (native SDK; reads `MISTRAL_API_KEY`, defaults to `mistral-large-latest`, supports sync / async / streaming) |
86-
| `LiteLLMProposer` | 100+ providers via LiteLLM (Bedrock, Vertex, Cohere, Mistral, ...) |
87-
71+
| Adapter | Covers |
72+
|---|---|
73+
| `ClaudeProposer` | Anthropic Claude (native SDK) |
74+
| `CohereProposer` | Cohere (native SDK) |
75+
| `OpenAIProposer` | OpenAI + **any OpenAI-compatible endpoint** (OpenRouter, Together, Groq, Fireworks, Anyscale, Perplexity, xAI, DeepSeek, Azure, Ollama, LM Studio, vLLM, llama.cpp server, ...) |
76+
| `GroqProposer` | Groq (OpenAI-compatible; reads `GROQ_API_KEY`, defaults to Llama 3.3 70B) |
77+
| `FireworksProposer` | Fireworks AI (Llama, Qwen, Mixtral, DeepSeek hosted models) |
78+
| `TogetherProposer` | Together AI (Llama, Qwen, DeepSeek hosted models) |
79+
| `GeminiProposer` | Google Gemini (native SDK) |
80+
| `MistralProposer` | Mistral AI (native SDK; reads `MISTRAL_API_KEY`, defaults to `mistral-large-latest`, supports sync / async / streaming) |
81+
| `LiteLLMProposer` | 100+ providers via LiteLLM (Bedrock, Vertex, Cohere, Mistral, ...) |
8882

8983
## Features
9084

9185
### Multi-turn repair with memory
92-
9386
Every proposal sees the history of *prior failed attempts* so the LLM can't repeat the same mistake. This is the single biggest quality win over naive retry.
9487

9588
### Verifiers: `verify=callable`
96-
9789
Catch bad *return values*, not just exceptions:
9890

9991
```python
@@ -104,7 +96,6 @@ def extract_price(text): ...
10496
If the predicate returns `False` or raises, self-heal treats it as a failure and repairs.
10597

10698
### Test-driven repair: `tests=[...]`
107-
10899
Give self-heal a test suite; it repairs until every test passes:
109100

110101
```python
@@ -116,7 +107,6 @@ def extract_price(text): ...
116107
```
117108

118109
### Async-native
119-
120110
The decorator auto-detects `async def` and awaits correctly; the LLM call runs in a thread pool so your event loop stays free.
121111

122112
```python
@@ -125,26 +115,21 @@ async def fetch_and_parse(url: str) -> dict: ...
125115
```
126116

127117
### Prompt customization: `prompt_extra="..."`
128-
129118
Append domain-specific instructions to every repair prompt. Useful for "always handle None inputs" or "use only the standard library."
130119

131120
### Bring your own LLM
132-
133121
Implement the `LLMProposer` Protocol (`def propose(self, system: str, user: str) -> str`) and pass it in.
134122

135123
### Repair cache: skip the LLM when you've seen it before
136-
137124
```python
138125
from self_heal import repair
139126

140127
@repair(cache_path=".self_heal_cache.db")
141128
def my_fn(...): ...
142129
```
143-
144130
First repair hits the LLM. Subsequent identical failures are served from SQLite (zero latency, zero cost). Keyed on source hash + failure signature with whitespace and memory-address normalization.
145131

146132
### Safety: AST rails + subprocess sandbox
147-
148133
Two independent layers. Combine them freely.
149134

150135
```python
@@ -158,13 +143,11 @@ def my_fn(...): ...
158143
@repair(safety=SafetyConfig(level="moderate", sandbox="subprocess"))
159144
def my_fn(...): ...
160145
```
161-
162146
`moderate` rejects proposals that call `eval` / `exec` / `os.system`, import `subprocess` / `socket` / `pickle` / `ctypes`, or touch `__globals__` / `__class__` / other escape hatches. `strict` additionally forbids any non-whitelisted import. The subprocess sandbox adds a real process boundary: args and return values are pickled over stdin/stdout, and the child inherits none of the caller's globals (proposals must be self-contained). See [Safety](#safety) for the full trust model.
163147

164148
> **Sandbox + imports.** When `sandbox="subprocess"` is active, the child runs with `python -I` in a fresh namespace. **The repaired function must import every module it uses at the top of the definition.** `import math` at the caller module scope does NOT reach the sandbox, so a proposal that references `math.sqrt` without a local `import math` raises `NameError` on the first call. `self-heal` already hints at this in the LLM prompt when sandbox is active, but if you're writing a proposer by hand the same rule applies.
165149
166150
### Progress callbacks
167-
168151
```python
169152
from self_heal import repair, RepairEvent
170153

@@ -174,11 +157,9 @@ def watch(event: RepairEvent):
174157
@repair(on_event=watch)
175158
def my_fn(...): ...
176159
```
177-
178160
Hooks fire on attempt start, failure, propose start/complete, install, cache hit/miss, safety violation, verify, and repair completion. Perfect for agent UIs and observability pipelines.
179161

180162
### Token streaming
181-
182163
When a callback is registered, self-heal streams LLM tokens through `propose_chunk` events as they arrive:
183164

184165
```python
@@ -191,15 +172,12 @@ def on_event(event: RepairEvent):
191172
@repair(on_event=on_event)
192173
def my_fn(...): ...
193174
```
194-
195-
All four built-in proposers stream natively via their SDKs. Custom proposers can implement `propose_stream(system, user) -> Iterator[str]` (and `apropose_stream` for async) to participate; those without streaming fall back to a single completion. See `[examples/streaming_progress.py](examples/streaming_progress.py)`.
175+
All four built-in proposers stream natively via their SDKs. Custom proposers can implement `propose_stream(system, user) -> Iterator[str]` (and `apropose_stream` for async) to participate; those without streaming fall back to a single completion. See [`examples/streaming_progress.py`](examples/streaming_progress.py).
196176

197177
### Native async proposers
198-
199178
`arun` prefers each SDK's native async client when the proposer provides `apropose`, falling back to `asyncio.to_thread(propose)` otherwise. All four built-in adapters ship with native async; custom proposers work either way.
200179

201180
### Resilience: retry on transient provider errors
202-
203181
Rate limits (429), service blips (502/503/504), and timeouts are common with LLM APIs. Pass a `RetryConfig` and self-heal will retry the proposer call with exponential backoff + jitter before giving up. Auth and validation errors are not retried.
204182

205183
```python
@@ -212,7 +190,6 @@ def my_fn(...): ...
212190
Defaults: 3 retries, 1s base delay, 2x backoff, ±25% jitter, capped at 30s. Retries do not count against `max_attempts` and each one fires a `transient_retry` event with `retry_attempt` and `retry_delay`. Set `retry_config=None` (the default) to disable retries.
213191

214192
### pytest plugin: `pytest --heal`
215-
216193
Mark any test with `@pytest.mark.heal(target="mymod.my_fn")`. When it fails with `--heal`, self-heal loads the target, repairs it using the test as verification, and prints the proposed diff at the end of the session.
217194

218195
```python
@@ -223,23 +200,19 @@ from mymod import extract_price
223200
def test_rupees():
224201
assert extract_price("₹1,299") == 1299.0
225202
```
226-
227203
```bash
228204
pytest --heal # print proposed fix, leave files untouched
229205
pytest --heal-apply # write the fix back to disk (creates a .py.heal-backup)
230206
pytest --heal-apply-force # also allow modification of git-dirty files
231207
```
232-
233208
`--heal-apply` uses libcst for AST-faithful replacement when installed, falling back to textual replacement. It refuses to modify files with uncommitted git changes unless `--heal-apply-force` is given.
234209

235210
### CLI: heal a function from the command line
236-
237211
```bash
238212
self-heal heal mymod.py::extract_price \
239213
--test tests/test_mymod.py::test_rupees \
240214
--apply
241215
```
242-
243216
Loads the function, runs self-heal with your pytest-style test as verification, prints a unified diff, and (with `--apply`) writes the fix back to the file.
244217

245218
## Why this exists
@@ -293,14 +266,12 @@ result = await loop.arun(my_async_fn, args=(...))
293266
## Using different providers
294267

295268
**Claude (default):**
296-
297269
```python
298270
@repair()
299271
def my_fn(...): ...
300272
```
301273

302274
**OpenAI:**
303-
304275
```python
305276
from self_heal.llm import OpenAIProposer
306277

@@ -309,7 +280,6 @@ def my_fn(...): ...
309280
```
310281

311282
**Gemini:**
312-
313283
```python
314284
from self_heal.llm import GeminiProposer
315285

@@ -318,7 +288,6 @@ def my_fn(...): ...
318288
```
319289

320290
**Mistral:**
321-
322291
```python
323292
from self_heal.llm import MistralProposer
324293

@@ -327,7 +296,6 @@ def my_fn(...): ...
327296
```
328297

329298
**Any OpenAI-compatible endpoint (OpenRouter, Groq, Ollama, ...):**
330-
331299
```python
332300
from self_heal.llm import OpenAIProposer
333301

@@ -346,7 +314,6 @@ OpenAIProposer(
346314
```
347315

348316
**LiteLLM catch-all (100+ providers):**
349-
350317
```python
351318
from self_heal.llm import LiteLLMProposer
352319

@@ -395,14 +362,14 @@ def price_from_text(text: str) -> float:
395362

396363
### Other frameworks (decorator stacking)
397364

398-
Examples in `[examples/`
365+
Examples in [`examples/`](examples):
399366

400-
- `[with_claude_agent_sdk.py](examples/with_claude_agent_sdk.py)`
401-
- `[with_openai_agents.py](examples/with_openai_agents.py)`
402-
- `[with_langchain.py](examples/with_langchain.py)`
403-
- `[with_crewai.py](examples/with_crewai.py)`
367+
- [`with_claude_agent_sdk.py`](examples/with_claude_agent_sdk.py)
368+
- [`with_openai_agents.py`](examples/with_openai_agents.py)
369+
- [`with_langchain.py`](examples/with_langchain.py)
370+
- [`with_crewai.py`](examples/with_crewai.py)
404371

405-
All examples are smoke-tested on every CI run via `[tests/test_examples_import.py](tests/test_examples_import.py)`.
372+
All examples are smoke-tested on every CI run via [`tests/test_examples_import.py`](tests/test_examples_import.py).
406373

407374
## Safety
408375

@@ -422,25 +389,25 @@ def parse_price(text: str) -> float:
422389

423390
## Roadmap
424391

425-
- v0.0.1: core repair loop + decorator + Claude backend
426-
- v0.0.2: OpenAI, Gemini, LiteLLM adapters; works with any LLM
427-
- v0.1.0: multi-turn memory, verifiers, test-driven repair, async, benchmark harness
428-
- v0.2.0: repair cache, AST safety rails, event callbacks, pytest plugin, CLI, extended benchmarks
429-
- v0.3.0: subprocess sandbox, `pytest --heal-apply`, QuixBugs benchmark, local-model sweep tooling
430-
- v0.4.0: streaming token events (`propose_chunk`), native async proposers (`apropose`) for all four adapters
431-
- **v0.4.1: sandbox preserves custom exceptions from proposals; `is_git_dirty` fails closed on timeout; Claude Agent SDK and LangChain/LangGraph first-class integrations**
432-
- v0.5: wasm sandbox, warm subprocess worker pool, first-class CrewAI / OpenAI Agents SDK integrations
433-
- v1.0: stable API + extended benchmark suite (HumanEval-Fix, Refactory)
392+
- [x] v0.0.1: core repair loop + decorator + Claude backend
393+
- [x] v0.0.2: OpenAI, Gemini, LiteLLM adapters; works with any LLM
394+
- [x] v0.1.0: multi-turn memory, verifiers, test-driven repair, async, benchmark harness
395+
- [x] v0.2.0: repair cache, AST safety rails, event callbacks, pytest plugin, CLI, extended benchmarks
396+
- [x] v0.3.0: subprocess sandbox, `pytest --heal-apply`, QuixBugs benchmark, local-model sweep tooling
397+
- [x] v0.4.0: streaming token events (`propose_chunk`), native async proposers (`apropose`) for all four adapters
398+
- [x] **v0.4.1: sandbox preserves custom exceptions from proposals; `is_git_dirty` fails closed on timeout; Claude Agent SDK and LangChain/LangGraph first-class integrations**
399+
- [ ] v0.5: wasm sandbox, warm subprocess worker pool, first-class CrewAI / OpenAI Agents SDK integrations
400+
- [ ] v1.0: stable API + extended benchmark suite (HumanEval-Fix, Refactory)
434401

435402
## Deeper docs
436403

437-
- `[docs/sandbox-threat-model.md](docs/sandbox-threat-model.md)`: what the subprocess sandbox protects against and what it does not. Read before running against untrusted inputs.
438-
- `[docs/custom-proposer.md](docs/custom-proposer.md)`: implementing the `LLMProposer` Protocol for an unsupported provider.
439-
- `[docs/faq.md](docs/faq.md)`: positioning, cost, safety, integrations, contribution.
404+
- [`docs/sandbox-threat-model.md`](docs/sandbox-threat-model.md): what the subprocess sandbox protects against and what it does not. Read before running against untrusted inputs.
405+
- [`docs/custom-proposer.md`](docs/custom-proposer.md): implementing the `LLMProposer` Protocol for an unsupported provider.
406+
- [`docs/faq.md`](docs/faq.md): positioning, cost, safety, integrations, contribution.
440407

441408
## Contributing
442409

443-
See `[CONTRIBUTING.md](CONTRIBUTING.md)` for the full guide: dev setup, everyday commands, how to add a new LLM proposer or benchmark task, and the PR checklist. Good first issues are tagged [here](https://github.com/Johin2/self-heal/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).
410+
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the full guide: dev setup, everyday commands, how to add a new LLM proposer or benchmark task, and the PR checklist. Good first issues are tagged [here](https://github.com/Johin2/self-heal/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).
444411

445412
## Development (quick start)
446413

@@ -455,7 +422,6 @@ ruff check .
455422
```
456423

457424
Run the benchmark locally:
458-
459425
```bash
460426
python benchmarks/run.py --proposer claude # uses ANTHROPIC_API_KEY
461427
python benchmarks/run.py --proposer openai # uses OPENAI_API_KEY
@@ -465,4 +431,4 @@ python benchmarks/run.py --suite quixbugs --proposer gemini # QuixBugs (40
465431

466432
## License
467433

468-
MIT
434+
MIT

src/self_heal/loop.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,8 @@ def _obtain_repair(self, ctx: _RunContext) -> str:
201201
)
202202

203203
proposer = self.proposer
204+
# The stream call is not retry-wrapped; a failed stream emits
205+
# stream_error and falls back to the retry-wrapped _run_propose below.
204206
if hasattr(proposer, "propose_stream") and self.on_event is not None:
205207
chunks: list[str] = []
206208
try:
@@ -248,6 +250,8 @@ async def _aobtain_repair(self, ctx: _RunContext) -> str:
248250
)
249251

250252
proposer = self.proposer
253+
# The stream call is not retry-wrapped; a failed stream emits
254+
# stream_error and falls back to the retry-wrapped _run_apropose below.
251255
if hasattr(proposer, "apropose_stream") and self.on_event is not None:
252256
chunks: list[str] = []
253257
try:

0 commit comments

Comments
 (0)