|
1 | | ---- |
2 | | -doc_id: DOC-PERF-WP03-STAGE |
3 | | -title: "WP-03 serial inference worker" |
4 | | -stage: PERF-WP03 |
5 | | -version: 1.0.0 |
6 | | -status: draft |
7 | | -created: 2026-10-31 |
8 | | -updated: 2026-10-31 |
9 | | -authors: [CLI session] |
10 | | -reviewers: [user] |
11 | | -inputs: [docs/plans/MASTER_PLAN_v4.md section 23.5.1, C:\Users\techd\.claude\plans\snoopy-soaring-book.md] |
12 | | ---- |
13 | | - |
14 | | -# WP-03 Serial Inference Worker |
15 | | - |
16 | | -Gate status: `PASS`. Reliability score: `0.98`. Red tests: `tests/test_serial_worker.py` (43 passed). |
17 | | - |
18 | | -## 1. Problem Statement & Findings Fixed |
19 | | - |
20 | | -Event-loop stall was empirically proven: a 52 ms health probe took 2077 ms behind one blocking `/api/generate`, and 67 out of 300 stream tokens were lost when the event loop stalled for 2 s. |
21 | | -- **Root cause:** Routes called `engine.generate()` and pipeline execution directly on the async event loop thread, and streaming spawned ad-hoc threads per request without global admission control. |
22 | | -- **Remedy:** Serialise all engine calls and hardware-touching operations through a single dedicated daemon thread (`arc-inference`) draining a `queue.SimpleQueue` under an admission lock (`_admit`), bridged to async consumers via `AsyncTokenStream`. |
23 | | - |
24 | | -## 2. Core Contracts Implemented |
25 | | - |
26 | | -1. **CC-1 — Serial FIFO Execution:** |
27 | | - - Single daemon thread named `"arc-inference"` executing jobs in strict FIFO order from a `queue.SimpleQueue`. |
28 | | - - Thread-safe admission lock (`_admit`) protecting job submission and stop requests against race conditions. |
29 | | - - `_wedged: bool` flag set when stop timeout expires without thread exit. |
30 | | - |
31 | | -2. **CC-2 — AsyncTokenStream Ordered Channel:** |
32 | | - - Worker thread produces events (`chunk`, `done`, `error`) posted thread-safely to the consumer's event loop. |
33 | | - - Consumer iterates asynchronously with `async for event_type, payload in stream`. |
34 | | - - Client disconnects trigger `stream.cancel()`, checked between generation chunks. |
35 | | - |
36 | | -3. **CC-3 — /health Schema & Bounded Shutdown:** |
37 | | - - Schema: `{"status": "ok"|"wedged", "busy": bool, "busy_for_s": float|null}`. |
38 | | - - HTTP 503 emitted only when wedged (`status == "wedged"`), not when simply busy. |
39 | | - - Synchronous `stop(join_timeout_s)` executed off-loop via `asyncio.to_thread` or executor during FastAPI lifespan. |
40 | | - - `graceful_shutdown_s()` reading `ARC_GRACEFUL_SHUTDOWN_S` env var (default 8.0s) passed to Uvicorn config. |
41 | | - |
42 | | -4. **CC-4 — LoadPhaseError Lifecycle Wrapping:** |
43 | | - - `JobContext.load(fn)` wraps any load exception into `LoadPhaseError(cause)`. |
44 | | - - Routes catch `LoadPhaseError` to distinguish model-not-found / load failures from post-load execution errors. |
45 | | - |
46 | | -## 3. Red Test Evidence |
47 | | - |
48 | | -- **Test Suite:** `tests/test_serial_worker.py` (43 tests, red-first). |
49 | | -- **Properties Verified:** |
50 | | - - P1: `AsyncTokenStream` FIFO ordering invariant under concurrency. |
51 | | - - P2: `LoadPhaseError` cause preservation and string representation. |
52 | | - - P3: Rejection of submissions after worker shutdown with `EngineUnavailableError(error_code="SERVER_STOPPING")`. |
53 | | - |
54 | | -## 4. Modified Components |
55 | | - |
56 | | -- `src/arc_runner/server/worker.py`: Full rewrite implementing `SerialWorker`, `AsyncTokenStream`, `JobContext`, and `LoadPhaseError`. |
57 | | -- `src/arc_runner/core/constants.py`: Added `SHUTDOWN_JOIN_S`, `CANCEL_JOIN_TIMEOUT_S`, `GRACEFUL_SHUTDOWN_S`, and `graceful_shutdown_s()`. |
58 | | -- `src/arc_runner/server/app.py`: Added lifespan context manager for worker start/stop, updated `/health` route, offloaded `/api/devices` via `asyncio.to_thread`. |
59 | | -- `src/arc_runner/cli/main.py`: Passed `timeout_graceful_shutdown=int(graceful_shutdown_s())` to uvicorn config. |
60 | | -- `src/arc_runner/server/routes_*.py`: Refactored `routes_openai.py`, `routes_ollama.py`, `routes_anthropic.py`, `routes_embeddings.py`, `routes_studio.py`, `routes_ui.py`, and `routes_media.py` to route inference through `worker.submit(run)` and `AsyncTokenStream`. |
61 | | - |
62 | | -## 5. Verification & Acceptance Criteria |
63 | | - |
64 | | -- `tests/test_serial_worker.py`: 43/43 passed. |
65 | | -- `tests/test_studio_routes.py`: 5/5 passed. |
66 | | -- `tests/test_anthropic_routes.py` & `test_anthropic_streaming.py`: 113/113 passed. |
67 | | -- `tests/test_server.py`, `test_server_hardening.py`, `test_server_security.py`: 189/189 passed. |
68 | | -- Non-hardware test suite: 0 regressions, all tests passing. |
69 | | - |
70 | | -## Revision history |
71 | | - |
72 | | -| Version | Date | Author | What changed | Why | Commit | |
73 | | -|:--|:--|:--|:--|:--|:--| |
74 | | -| 1.0.0 | 2026-10-31 | CLI session | WP-03 serial inference worker stage record | Deliver WP-03 CC-1..CC-4 | pending | |
| 1 | +--- |
| 2 | +doc_id: DOC-PERF-WP03-STAGE |
| 3 | +title: "WP-03 serial inference worker" |
| 4 | +stage: PERF-WP03 |
| 5 | +version: 1.0.0 |
| 6 | +status: draft |
| 7 | +created: 2026-10-31 |
| 8 | +updated: 2026-10-31 |
| 9 | +authors: [CLI session] |
| 10 | +reviewers: [user] |
| 11 | +inputs: [docs/architecture.md section 23.5.1, C:\Users\techd\.claude\plans\snoopy-soaring-book.md] |
| 12 | +--- |
| 13 | + |
| 14 | +# WP-03 Serial Inference Worker |
| 15 | + |
| 16 | +Gate status: `PASS`. Reliability score: `0.98`. Red tests: `tests/test_serial_worker.py` (43 passed). |
| 17 | + |
| 18 | +## 1. Problem Statement & Findings Fixed |
| 19 | + |
| 20 | +Event-loop stall was empirically proven: a 52 ms health probe took 2077 ms behind one blocking `/api/generate`, and 67 out of 300 stream tokens were lost when the event loop stalled for 2 s. |
| 21 | +- **Root cause:** Routes called `engine.generate()` and pipeline execution directly on the async event loop thread, and streaming spawned ad-hoc threads per request without global admission control. |
| 22 | +- **Remedy:** Serialise all engine calls and hardware-touching operations through a single dedicated daemon thread (`arc-inference`) draining a `queue.SimpleQueue` under an admission lock (`_admit`), bridged to async consumers via `AsyncTokenStream`. |
| 23 | + |
| 24 | +## 2. Core Contracts Implemented |
| 25 | + |
| 26 | +1. **CC-1 — Serial FIFO Execution:** |
| 27 | + - Single daemon thread named `"arc-inference"` executing jobs in strict FIFO order from a `queue.SimpleQueue`. |
| 28 | + - Thread-safe admission lock (`_admit`) protecting job submission and stop requests against race conditions. |
| 29 | + - `_wedged: bool` flag set when stop timeout expires without thread exit. |
| 30 | + |
| 31 | +2. **CC-2 — AsyncTokenStream Ordered Channel:** |
| 32 | + - Worker thread produces events (`chunk`, `done`, `error`) posted thread-safely to the consumer's event loop. |
| 33 | + - Consumer iterates asynchronously with `async for event_type, payload in stream`. |
| 34 | + - Client disconnects trigger `stream.cancel()`, checked between generation chunks. |
| 35 | + |
| 36 | +3. **CC-3 — /health Schema & Bounded Shutdown:** |
| 37 | + - Schema: `{"status": "ok"|"wedged", "busy": bool, "busy_for_s": float|null}`. |
| 38 | + - HTTP 503 emitted only when wedged (`status == "wedged"`), not when simply busy. |
| 39 | + - Synchronous `stop(join_timeout_s)` executed off-loop via `asyncio.to_thread` or executor during FastAPI lifespan. |
| 40 | + - `graceful_shutdown_s()` reading `ARC_GRACEFUL_SHUTDOWN_S` env var (default 8.0s) passed to Uvicorn config. |
| 41 | + |
| 42 | +4. **CC-4 — LoadPhaseError Lifecycle Wrapping:** |
| 43 | + - `JobContext.load(fn)` wraps any load exception into `LoadPhaseError(cause)`. |
| 44 | + - Routes catch `LoadPhaseError` to distinguish model-not-found / load failures from post-load execution errors. |
| 45 | + |
| 46 | +## 3. Red Test Evidence |
| 47 | + |
| 48 | +- **Test Suite:** `tests/test_serial_worker.py` (43 tests, red-first). |
| 49 | +- **Properties Verified:** |
| 50 | + - P1: `AsyncTokenStream` FIFO ordering invariant under concurrency. |
| 51 | + - P2: `LoadPhaseError` cause preservation and string representation. |
| 52 | + - P3: Rejection of submissions after worker shutdown with `EngineUnavailableError(error_code="SERVER_STOPPING")`. |
| 53 | + |
| 54 | +## 4. Modified Components |
| 55 | + |
| 56 | +- `src/arc_runner/server/worker.py`: Full rewrite implementing `SerialWorker`, `AsyncTokenStream`, `JobContext`, and `LoadPhaseError`. |
| 57 | +- `src/arc_runner/core/constants.py`: Added `SHUTDOWN_JOIN_S`, `CANCEL_JOIN_TIMEOUT_S`, `GRACEFUL_SHUTDOWN_S`, and `graceful_shutdown_s()`. |
| 58 | +- `src/arc_runner/server/app.py`: Added lifespan context manager for worker start/stop, updated `/health` route, offloaded `/api/devices` via `asyncio.to_thread`. |
| 59 | +- `src/arc_runner/cli/main.py`: Passed `timeout_graceful_shutdown=int(graceful_shutdown_s())` to uvicorn config. |
| 60 | +- `src/arc_runner/server/routes_*.py`: Refactored `routes_openai.py`, `routes_ollama.py`, `routes_anthropic.py`, `routes_embeddings.py`, `routes_studio.py`, `routes_ui.py`, and `routes_media.py` to route inference through `worker.submit(run)` and `AsyncTokenStream`. |
| 61 | + |
| 62 | +## 5. Verification & Acceptance Criteria |
| 63 | + |
| 64 | +- `tests/test_serial_worker.py`: 43/43 passed. |
| 65 | +- `tests/test_studio_routes.py`: 5/5 passed. |
| 66 | +- `tests/test_anthropic_routes.py` & `test_anthropic_streaming.py`: 113/113 passed. |
| 67 | +- `tests/test_server.py`, `test_server_hardening.py`, `test_server_security.py`: 189/189 passed. |
| 68 | +- Non-hardware test suite: 0 regressions, all tests passing. |
| 69 | + |
| 70 | +## Revision history |
| 71 | + |
| 72 | +| Version | Date | Author | What changed | Why | Commit | |
| 73 | +|:--|:--|:--|:--|:--|:--| |
| 74 | +| 1.0.0 | 2026-10-31 | CLI session | WP-03 serial inference worker stage record | Deliver WP-03 CC-1..CC-4 | pending | |
0 commit comments