Merge pull request #14 from pankaj4u4m/telemetry

Telemetry

Author: pankaj4u4m

Makefile

@@ -87,7 +87,7 @@ pre-commit: format lint typecheck test ## Format, lint, typecheck, and test
 # --------------------------------------------------------------------------- #
 
 benchmark: ## Run the full benchmark suite
-	LOCAL=1 atelier benchmark-full --json
+	LOCAL=1 atelier benchmark full --json
 
 bench-savings: ## Run the context-savings benchmark
 	LOCAL=1 uv run python -m benchmarks.swe.savings_bench --json

QUICK_REFERENCE.md

@@ -167,7 +167,7 @@ atelier verify rubric_shopify_publish --checks '{"check_1": true, ...}'
 atelier block list --domain beseam.shopify.publish
 atelier eval list
 atelier eval run <case_id>
-atelier benchmark --apply
+atelier benchmark run
 atelier analyze-failures
 atelier savings
 atelier tool-mode set shadow

benchmarks/swe/README.md

@@ -14,7 +14,7 @@ events, rescue events, rubric verdicts).
 ```
 benchmarks/swe/
 ├── README.md          ← you are here
-├── run_swe_bench.py   ← Click entry: `atelier-bench swe …`
+├── run_swe_bench.py   ← Click entry: `atelier benchmark swe ...`
 ├── config.py          ← strict pydantic config schema
 ├── datasets.py        ← swe_bench_lite / verified / mock / custom JSONL
 ├── task_runner.py     ← one (task × mode × attempt) → metrics row
@@ -45,14 +45,14 @@ benchmarks/swe/
 `show-modes` prints the matrix at runtime:
 
 ```bash
-uv run atelier-bench swe show-modes
+uv run atelier benchmark swe show-modes
 ```
 
 ## Quickstart (offline, no API keys)
 
 ```bash
 # 20-task mock run (uses built-in mock dataset + MockAgent)
-uv run atelier-bench swe run --config benchmarks/swe/configs/lite_20.yaml
+uv run atelier benchmark swe run --config benchmarks/swe/configs/lite_20.yaml
 
 # Output:
 #   benchmarks/swe/outputs/lite_20/<TIMESTAMP>/
@@ -64,10 +64,10 @@ uv run atelier-bench swe run --config benchmarks/swe/configs/lite_20.yaml
 #     report.json
 
 # Score predictions (mock evaluator if `swebench` not installed)
-uv run atelier-bench swe evaluate --run-dir benchmarks/swe/outputs/lite_20/<TS> --mock
+uv run atelier benchmark swe evaluate --run-dir benchmarks/swe/outputs/lite_20/<TS> --mock
 
 # Re-render report from existing metrics
-uv run atelier-bench swe report --run-dir benchmarks/swe/outputs/lite_20/<TS>
+uv run atelier benchmark swe report --run-dir benchmarks/swe/outputs/lite_20/<TS>
 ```
 
 ## Make targets
@@ -116,7 +116,7 @@ This is the schema accepted by
 
 ```bash
 uv pip install swebench
-uv run atelier-bench swe evaluate --run-dir benchmarks/swe/outputs/lite_20/<TS>
+uv run atelier benchmark swe evaluate --run-dir benchmarks/swe/outputs/lite_20/<TS>
 ```
 
 If `swebench` is missing the command prints exact install instructions and

benchmarks/swe/__init__.py

@@ -8,7 +8,7 @@
 Public entry points:
 
 * :func:`benchmarks.swe.run_swe_bench.cli` — Click command group exposed
-  as the ``atelier-bench`` console script.
+  as ``atelier benchmark swe`` subcommands.
 * :class:`benchmarks.swe.config.BenchConfig`
 * :class:`benchmarks.swe.modes.Mode`
 

benchmarks/swe/run_swe_bench.py

@@ -1,4 +1,4 @@
-"""``atelier-bench swe`` — Click command group entry point.
+"""``atelier benchmark swe`` — Click command group entry point.
 
 Subcommands:
     run                     run the harness end-to-end against a config
@@ -41,7 +41,7 @@ def swe() -> None:
 
 @click.group(context_settings={"help_option_names": ["-h", "--help"]})
 def cli() -> None:
-    """atelier-bench — benchmarking harnesses for Atelier."""
+    """SWE-bench benchmarking harnesses for Atelier."""
 
 
 cli.add_command(swe)
@@ -219,8 +219,8 @@ def cmd_measure_context_savings(suite_path: str | None, emit_json: bool) -> None
         raise click.ClickException(f"context savings {result.reduction_pct:.2f}% is below the 50% CI gate")
 
 
-def main() -> None:  # console-script entry point
-    cli()
+def main() -> None:
+    raise click.ClickException("atelier-bench has been removed; use `atelier benchmark swe ...`")
 
 
 if __name__ == "__main__":

benchmarks/swe/savings_replay.py

@@ -478,9 +478,10 @@ def _persist_result(db_path: Path, result: ReplayResult, completed_at: datetime,
             conn.execute(
                 """
                 INSERT INTO benchmark_prompt_result (
-                    id, session_id, prompt_id, task_type, baseline_input_tokens,
-                    optimized_input_tokens, reduction_pct, lever_attribution_json
-                ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                    id, session_id, prompt_id, task_type, input_tokens_baseline,
+                    input_tokens_optimized, reduction_pct, duration_ms, created_at,
+                    baseline_input_tokens, optimized_input_tokens, lever_attribution_json
+                ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                 """,
                 (
                     f"bpr-{make_uuid7()}",
@@ -490,6 +491,10 @@ def _persist_result(db_path: Path, result: ReplayResult, completed_at: datetime,
                     prompt.baseline_input_tokens,
                     prompt.optimized_input_tokens,
                     prompt.reduction_pct,
+                    0,
+                    completed_at.isoformat(),
+                    prompt.baseline_input_tokens,
+                    prompt.optimized_input_tokens,
                     json.dumps({prompt.lever: prompt.tokens_saved}, sort_keys=True),
                 ),
             )
@@ -530,9 +535,10 @@ def _persist_paired_command_result(
             conn.execute(
                 """
                 INSERT INTO benchmark_prompt_result (
-                    id, session_id, prompt_id, task_type, baseline_input_tokens,
-                    optimized_input_tokens, reduction_pct, lever_attribution_json
-                ) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+                    id, session_id, prompt_id, task_type, input_tokens_baseline,
+                    input_tokens_optimized, reduction_pct, duration_ms, created_at,
+                    baseline_input_tokens, optimized_input_tokens, lever_attribution_json
+                ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                 """,
                 (
                     f"bpr-{make_uuid7()}",
@@ -542,6 +548,10 @@ def _persist_paired_command_result(
                     prompt.baseline.total_tokens,
                     prompt.atelier.total_tokens,
                     prompt.reduction_pct,
+                    prompt.baseline.elapsed_ms + prompt.atelier.elapsed_ms,
+                    completed_at.isoformat(),
+                    prompt.baseline.total_tokens,
+                    prompt.atelier.total_tokens,
                     json.dumps(
                         {
                             "tokens_saved": prompt.tokens_saved,

benchmarks/swe/swebench_eval.py

@@ -26,7 +26,7 @@
     "[swebench] official evaluator not installed.\n"
     "  pip install swebench\n"
     "  # or, with uv:  uv pip install swebench\n"
-    "Then re-run:  uv run atelier-bench swe evaluate --run-dir <dir>\n"
+    "Then re-run:  uv run atelier benchmark swe evaluate --run-dir <dir>\n"
     "Reference: https://github.com/princeton-nlp/SWE-bench"
 )
 

docs/benchmarks/phase7-2026-04-29.md

@@ -62,12 +62,12 @@ Round-by-round breakdown for `pdp` (representative; all 5 tasks behave identical
 cd atelier
 rm -rf /tmp/atelier-bench && mkdir -p /tmp/atelier-bench
 atelier --root /tmp/atelier-bench init
-atelier --root /tmp/atelier-bench benchmark --rounds 5
+atelier --root /tmp/atelier-bench benchmark run --rounds 5
 atelier --root /tmp/atelier-bench savings-detail
 atelier --root /tmp/atelier-bench savings
 ```
 
-For machine-readable output: `atelier --root /tmp/atelier-bench benchmark --rounds 5 --json`.
+For machine-readable output: `atelier --root /tmp/atelier-bench benchmark run --rounds 5 --json`.
 
 ---
 

docs/core/benchmarking.md

@@ -4,9 +4,9 @@ Atelier benchmarks runtime efficiency against baseline host behavior.
 
 ## CLI
 
-- `atelier benchmark-core`
-- `atelier benchmark-runtime`
-- `atelier benchmark-host`
+- `atelier benchmark core`
+- `atelier benchmark runtime`
+- `atelier benchmark hosts`
 
 ## Runtime metrics
 
@@ -38,6 +38,6 @@ Interpretation guidance:
 
 ## Notes
 
-`benchmark-runtime` emits capability-focused efficiency metrics.
-`benchmark-core` runs prompt/task benchmark rounds.
-`benchmark-host` verifies host integration health.
+`benchmark runtime` emits capability-focused efficiency metrics.
+`benchmark core` runs prompt/task benchmark rounds.
+`benchmark hosts` verifies host integration health.

docs/engineering/dogfooding.md

@@ -141,7 +141,7 @@ LOCAL=1 uv run python -m pytest tests/test_swe_benchmark_harness.py -q
 
 ```bash
 atelier --root .atelier pack install src/atelier/packs/official/atelier-pack-coding-general --json
-atelier --root .atelier benchmark-packs --json
+atelier --root .atelier benchmark packs --json
 ```
 
 **Expected:** install succeeds and benchmark reports baseline vs host+core vs host+core+pack metrics.