feat(etl): add agent-native observable pipelines

Add whole-run and micro-batch async ETL scaffolds with atomic local run state, machine-readable receipts, explicit dry-run semantics, and agent-facing progress.

Document the architecture boundaries, examples, import contracts, and failure/cancellation observation behavior. Closes #147.

Author: keli-wen

.agents/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers contributor setup (environment + hooks), filing issues, commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when setting up as a contributor, filing an issue, committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers contributor setup (environment + hooks), filing issues, commit format, pull request format, and component development across quantmind/ modules (etl, knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when setting up as a contributor, filing an issue, committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.agents/skills/quantmind-dev/references/develop-components.md

@@ -39,14 +39,34 @@ apply throughout.
 
 | Module | May import from `quantmind.*` |
 |--------|-------------------------------|
+| `quantmind/etl/` | nothing (independent leaf) |
 | `quantmind/utils/` | nothing (leaf) |
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
 | `quantmind/rag/` | `preprocess` only |
 | `quantmind/library/` | `knowledge` only |
 | `quantmind/mind/` | `knowledge`, `configs`, `utils` (retrieval is library-free; the `library` edge is reserved for the future collection path, not single-tree `retrieve`) |
-| `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
+| `quantmind/flows/`, `quantmind/magic.py` | apex — may import domain layers above, but not the independent `etl` scaffold |
+
+### `quantmind/etl/` — observable whole-run and micro-batch ETL
+
+- Bind exactly three async stage callables to `ETLPipeline` for one whole-run
+  delivery. Use the parallel `BatchETLPipeline` when an async producer yields
+  business batches that each pass through transform and load. Never switch
+  execution shape by inspecting a callable's return value, and never hide batch
+  loads inside a whole-run transform.
+- Use composition rather than an ABC, subclass tree, or inheritance between the
+  two pipeline classes. Keep batch execution strictly serial unless a later
+  observation contract explicitly represents simultaneously active stages.
+- Keep it independent of every other `quantmind.*` package. Existing flows do
+  not inherit it; their pure `input → artifact` contract remains unchanged.
+- Report only real completed work through `PipelineContext.progress()`. In batch
+  mode, only a load that returns successfully increments the completed-batch
+  count; partial-write safety remains the business load's responsibility. The
+  scaffold owns its local lifecycle snapshots; do not add custom run-state
+  files, a CLI, heartbeat, scheduler, retry policy, checkpoint/resume, or
+  workflow engine. See `contexts/design/operations/etl.md`.
 
 ### `quantmind/knowledge/` — data standard
 
@@ -144,10 +164,13 @@ apply throughout.
 A public operation is complete only when all of these agree:
 
 1. A stage and name consistent with `contexts/design/operations/naming.md`.
-2. Typed input and config models, exported from `quantmind.configs`.
+2. Typed input and config models, when the operation has them, exported from
+   the canonical owning package (`quantmind.configs` for flow configs,
+   `quantmind.etl` for ETL run contracts, or another explicit owner).
 3. One intent-oriented async function, small service class, or document-scoped
-   handle exported from `quantmind.flows`, with its result contract exported
-   from the canonical owning layer.
+   handle exported from its canonical owning package (`quantmind.flows`,
+   `quantmind.etl`, `quantmind.library`, etc.), with its result contract
+   exported from the same owning layer.
 4. Offline success and failure tests for the public callable, plus a
    magic-introspection test when a function follows the `(input, *, cfg)`
    convention.

.claude/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers contributor setup (environment + hooks), filing issues, commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when setting up as a contributor, filing an issue, committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers contributor setup (environment + hooks), filing issues, commit format, pull request format, and component development across quantmind/ modules (etl, knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when setting up as a contributor, filing an issue, committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.claude/skills/quantmind-dev/references/develop-components.md

@@ -39,14 +39,34 @@ apply throughout.
 
 | Module | May import from `quantmind.*` |
 |--------|-------------------------------|
+| `quantmind/etl/` | nothing (independent leaf) |
 | `quantmind/utils/` | nothing (leaf) |
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
 | `quantmind/rag/` | `preprocess` only |
 | `quantmind/library/` | `knowledge` only |
 | `quantmind/mind/` | `knowledge`, `configs`, `utils` (retrieval is library-free; the `library` edge is reserved for the future collection path, not single-tree `retrieve`) |
-| `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
+| `quantmind/flows/`, `quantmind/magic.py` | apex — may import domain layers above, but not the independent `etl` scaffold |
+
+### `quantmind/etl/` — observable whole-run and micro-batch ETL
+
+- Bind exactly three async stage callables to `ETLPipeline` for one whole-run
+  delivery. Use the parallel `BatchETLPipeline` when an async producer yields
+  business batches that each pass through transform and load. Never switch
+  execution shape by inspecting a callable's return value, and never hide batch
+  loads inside a whole-run transform.
+- Use composition rather than an ABC, subclass tree, or inheritance between the
+  two pipeline classes. Keep batch execution strictly serial unless a later
+  observation contract explicitly represents simultaneously active stages.
+- Keep it independent of every other `quantmind.*` package. Existing flows do
+  not inherit it; their pure `input → artifact` contract remains unchanged.
+- Report only real completed work through `PipelineContext.progress()`. In batch
+  mode, only a load that returns successfully increments the completed-batch
+  count; partial-write safety remains the business load's responsibility. The
+  scaffold owns its local lifecycle snapshots; do not add custom run-state
+  files, a CLI, heartbeat, scheduler, retry policy, checkpoint/resume, or
+  workflow engine. See `contexts/design/operations/etl.md`.
 
 ### `quantmind/knowledge/` — data standard
 
@@ -144,10 +164,13 @@ apply throughout.
 A public operation is complete only when all of these agree:
 
 1. A stage and name consistent with `contexts/design/operations/naming.md`.
-2. Typed input and config models, exported from `quantmind.configs`.
+2. Typed input and config models, when the operation has them, exported from
+   the canonical owning package (`quantmind.configs` for flow configs,
+   `quantmind.etl` for ETL run contracts, or another explicit owner).
 3. One intent-oriented async function, small service class, or document-scoped
-   handle exported from `quantmind.flows`, with its result contract exported
-   from the canonical owning layer.
+   handle exported from its canonical owning package (`quantmind.flows`,
+   `quantmind.etl`, `quantmind.library`, etc.), with its result contract
+   exported from the same owning layer.
 4. Offline success and failure tests for the public callable, plus a
    magic-introspection test when a function follows the `(input, *, cfg)`
    convention.

.gitignore

@@ -29,12 +29,14 @@ docs/superpowers/
 
 # Coverage artifacts (generated by pytest --cov)
 .coverage
+.coverage.*
 htmlcov/
 coverage.xml
 
 # macOS
 .DS_Store
 
 # Ephemeral local scratch (temp dirs, e2e harness output)
+.quant-mind/
 temp/
 tmp/

AGENTS.md

@@ -46,6 +46,7 @@ The canonical, always-current statement lives in
 
 | Module | Role |
 |--------|------|
+| `quantmind/etl/` | Stdlib-only, observable whole-run and micro-batch `extract → transform → load` authoring scaffolds — independent leaf |
 | `quantmind/knowledge/` | Pydantic data standard (`FlattenKnowledge` / `TreeKnowledge` / `GraphKnowledge`) — dependency leaf |
 | `quantmind/library/` | Local persistence and semantic retrieval for canonical knowledge — depends only on `knowledge` |
 | `quantmind/configs/` | Operation cfg + typed input models or unions (`BaseFlowCfg`, `NewsWindow`, `PaperInput`) — depends only on `knowledge` |
@@ -131,6 +132,13 @@ the user explicitly authorizes it — fix the underlying issue instead.
    `BaseKnowledge`. Accept modest redundancy to keep artifacts self-contained.
    Half-finished intermediates stay component seams, not public flows. See
    `contexts/design/operations/orchestration.md`.
+10. **ETL execution stays separate from flows** — use
+    `quantmind.etl.ETLPipeline` for one whole-run delivery and
+    `BatchETLPipeline` for repeated business-batch deliveries. Call
+    `ctx.progress()` in long loops so the next Agent can observe real completed
+    work; do not write custom run-state files. Both are composition-based
+    authoring scaffolds, not base classes for `quantmind.flows`. See
+    `contexts/design/operations/etl.md`.
 
 ## Tests and Examples
 

contexts/CONTEXT_MAP.md

@@ -28,6 +28,7 @@ contexts/
 │   ├── library/local.md        ← LocalKnowledgeLibrary storage and retrieval
 │   ├── mind/retrieval.md       ← page-preserving structure tree + agentic retrieval
 │   ├── operations/
+│   │   ├── etl.md              ← whole-run vs micro-batch ETL execution and observation
 │   │   ├── naming.md           ← public operation naming rules
 │   │   └── orchestration.md    ← pipelines vs components (altitude)
 │   ├── preprocess/pdf.md       ← page-aware ParsedDocument

contexts/design/README.md

@@ -28,6 +28,7 @@ This directory records QuantMind engineering decisions. Use it to understand whi
 | Mind | [Build and retrieve from a page-preserving structure tree](mind/retrieval.md) |
 | Operations | [Public operation naming](operations/naming.md) |
 | Operations | [Orchestration and construction altitude](operations/orchestration.md) |
+| Operations | [Observable whole-run and micro-batch ETL](operations/etl.md) |
 | Utils | [Cross-provider structured output](utils/structured_output.md) |
 | Utils | [Collect per-run token and timing usage from SDK traces](utils/usage.md) |