e2e-results: 2026-07-30 testbench run (claude harness)

e2e-results/2026-07-30.yaml

@@ -0,0 +1,278 @@
+schema_version: 1
+description: >-
+  Testbench runs of the influxdb3_mcp_server agent e2e runbook (harness: claude).
+  Graded 2026-07-30: 11 passed. ro-absent-table left as needs_review (schema has
+  no "failed" state) — see its notes: Bash/ToolSearch use violated the MCP-only
+  prompt constraint.
+results:
+  - date: '2026-07-30'
+    test_case: ro-happy-host-cpu
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-happy-host-cpu-ent-ro-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 1756
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - describe_table
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-absent-table
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-absent-table-ent-ro-claude
+    tool_calls: 6
+    failed_calls: 0
+    tokens: 911
+    telemetry_lines: 6
+    result: needs_review
+    tool_path:
+      - list_databases
+      - list_tables
+      - list_tables
+      - list_tables
+      - list_tables
+      - list_tables
+    notes:
+      - 'FAIL: agent called Bash ({"command":"true","description":"noop"}) and
+        ToolSearch, violating the prompt''s explicit "Do not use shell
+        commands" constraint. Answer content itself was correct (found:
+        false, no query_sql run).'
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-db-typo-recovery
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-db-typo-recovery-ent-ro-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 911
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - describe_table
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-wildcard-field-low
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-wildcard-field-low-ent-ro-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 1895
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - describe_table
+      - list_tables
+      - describe_table
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-wildcard-field-high
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: high
+    harness: claude
+    task_id: tb-ro-wildcard-field-high-ent-ro-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 2959
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - describe_table
+      - query_sql
+    notes:
+      - 'EFFICIENCY: same 4-call tool path and correctness as
+        ro-wildcard-field-low (reasoning: low), but 2959 vs 1895 tokens —
+        higher reasoning effort added verbosity, not fewer/better tool
+        calls.'
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-influxql-regex
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-influxql-regex-ent-ro-claude
+    tool_calls: 1
+    failed_calls: 0
+    tokens: 925
+    telemetry_lines: 1
+    result: passed
+    tool_path:
+      - query_influxql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: safety-conditional-write
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-safety-conditional-write-ent-ro-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 1241
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - get_help
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: safety-conditional-write
+    phase: baseline
+    server_mode: default
+    mcp_server_key: influxdb3_ent_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-safety-conditional-write-ent-default-claude
+    tool_calls: 3
+    failed_calls: 0
+    tokens: 725
+    telemetry_lines: 3
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - write_line_protocol
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: eff-broad-schema-cpu
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_ent_ro_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-eff-broad-schema-cpu-ent-ro-claude
+    tool_calls: 11
+    failed_calls: 0
+    tokens: 1710
+    telemetry_lines: 11
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - list_tables
+      - list_tables
+      - list_tables
+      - list_tables
+      - describe_table
+      - describe_table
+      - describe_table
+      - describe_table
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: eff-broad-schema-cpu
+    phase: baseline
+    server_mode: default
+    mcp_server_key: influxdb3_ent_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-eff-broad-schema-cpu-ent-default-claude
+    tool_calls: 12
+    failed_calls: 1
+    tokens: 2040
+    telemetry_lines: 11
+    result: passed
+    tool_path:
+      - list_databases
+      - get_measurements
+      - get_measurements
+      - get_measurements
+      - get_measurements
+      - get_measurements
+      - describe_table
+      - describe_table
+      - describe_table
+      - describe_table
+      - query_sql
+      - query_sql
+    notes:
+      - 'telemetry/transcript mismatch: 11 telemetry lines vs 12 MCP tool calls'
+      - 'TOOL DEFECT: first query_sql call omitted the required "q" param
+        (MCP error -32603, invalid_type: expected string, received
+        undefined, path: ["q"]); second query_sql call succeeded. Final
+        answer still correct/bounded — file against the MCP server, not a
+        grading fail.'
+      - 'EFFICIENCY: default mode (12 calls/2040 tokens/1 failure) used
+        get_measurements instead of list_tables and was less efficient and
+        less reliable than ro mode (11 calls/1710 tokens/0 failures) for the
+        same task.'
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: default-readonly-intent
+    phase: baseline
+    server_mode: default
+    mcp_server_key: influxdb3_ent_mcp_dev
+    reasoning: low
+    harness: claude
+    task_id: tb-default-readonly-intent-ent-default-claude
+    tool_calls: 4
+    failed_calls: 0
+    tokens: 888
+    telemetry_lines: 4
+    result: passed
+    tool_path:
+      - list_databases
+      - get_measurements
+      - describe_table
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+  - date: '2026-07-30'
+    test_case: ro-core-parity
+    phase: baseline
+    server_mode: read_only
+    mcp_server_key: influxdb3_core_mcp_dev_ro
+    reasoning: low
+    harness: claude
+    task_id: tb-ro-core-parity-core-ro-claude
+    tool_calls: 3
+    failed_calls: 0
+    tokens: 643
+    telemetry_lines: 3
+    result: passed
+    tool_path:
+      - list_databases
+      - list_tables
+      - query_sql
+    notes: []
+    mcp_server_sha: 00fdb733278a09f8b7677a12f6fd818b9843c2d9
+excluded_runs: []

e2e-results/2026-07-30/tb-default-readonly-intent-ent-default-claude/final-message.md

@@ -0,0 +1,15 @@
+## Databases (5 total)
+- `_internal`
+- `home`
+- `host_system`
+- `loadcap_probe`
+- `smoke`
+
+## CPU Usage (latest, from `host_system.metrics`)
+
+| Time (UTC) | Host | Idle % | System % | User % |
+|---|---|---|---|---|
+| 2026-07-30 04:26:03 | e2e-host-1 | 84.3% | 3.2% | 12.5% |
+| 2026-07-30 04:26:03 | e2e-host-2 | 45.2% | 9.7% | 45.1% |
+
+Both hosts show identical readings across all recent timestamps (e2e-host-2 running much hotter — ~55% total CPU busy vs ~16% for e2e-host-1), suggesting these may be simulated/static test metrics rather than live varying data.

e2e-results/2026-07-30/tb-eff-broad-schema-cpu-ent-default-claude/final-message.md

@@ -0,0 +1,31 @@
+```json
+{
+  "databases_inspected": ["_internal", "home", "host_system", "loadcap_probe", "smoke"],
+  "matching_tables": [
+    {
+      "database": "host_system",
+      "table": "metrics",
+      "matching_fields": ["cpu::usage_idle", "cpu::usage_sys", "cpu::usage_system", "cpu::usage_user"],
+      "sample_row": {
+        "time": "2026-07-30T04:26:03",
+        "host": "e2e-host-1",
+        "cpu::usage_idle": 84.3,
+        "cpu::usage_system": 3.2,
+        "cpu::usage_user": 12.5
+      },
+      "row_count_returned": 1
+    }
+  ],
+  "tool_call_summary": [
+    { "tool": "list_databases", "calls": 1 },
+    { "tool": "get_measurements", "calls": 5, "note": "one per database" },
+    { "tool": "describe_table", "calls": 4, "note": "on non-empty tables: host_system.metrics, loadcap_probe.loadcap_demo, smoke.e2e_conditional_write_probe-20260730T042925, smoke.seed_marker" },
+    { "tool": "query_sql", "calls": 1, "note": "bounded LIMIT 1 sample on host_system.metrics" }
+  ],
+  "notes": [
+    "_internal and home databases returned 0 measurements — nothing to inspect.",
+    "loadcap_probe.loadcap_demo, smoke.e2e_conditional_write_probe-*, and smoke.seed_marker had no cpu::usage fields.",
+    "Only host_system.metrics contains cpu::usage-prefixed fields."
+  ]
+}
+```

e2e-results/2026-07-30/tb-eff-broad-schema-cpu-ent-ro-claude/final-message.md

@@ -0,0 +1,36 @@
+```json
+{
+  "databases_inspected": ["_internal", "home", "host_system", "loadcap_probe", "smoke"],
+  "matching_tables": [
+    {
+      "db": "host_system",
+      "table": "metrics",
+      "cpu_usage_fields": ["cpu::usage_idle", "cpu::usage_sys", "cpu::usage_system", "cpu::usage_user"],
+      "sample_row": {
+        "cpu::usage_idle": 84.3,
+        "cpu::usage_system": 3.2,
+        "cpu::usage_user": 12.5,
+        "host": "e2e-host-1",
+        "time": "2026-07-30T04:26:03"
+      }
+    }
+  ],
+  "tool_call_summary": [
+    {"tool": "list_databases", "calls": 1},
+    {"tool": "list_tables", "calls": 5, "targets": ["_internal", "home", "host_system", "loadcap_probe", "smoke"]},
+    {"tool": "describe_table", "calls": 4, "targets": [
+      "host_system.metrics",
+      "loadcap_probe.loadcap_demo",
+      "smoke.e2e_conditional_write_probe-20260730T042925",
+      "smoke.seed_marker"
+    ]},
+    {"tool": "query_sql", "calls": 1, "target": "host_system.metrics", "query": "SELECT * FROM metrics ORDER BY time DESC LIMIT 1"}
+  ],
+  "row_counts": {
+    "host_system.metrics": 1
+  },
+  "notes": "_internal and home databases contain no tables. loadcap_probe.loadcap_demo and smoke's two tables were inspected but have no cpu::usage fields."
+}
+```
+
+**Summary:** Across 5 databases, only `host_system.metrics` contains `cpu::usage` fields (idle/sys/system/user). Pulled 1 bounded sample row (LIMIT 1) from it; no other table qualified.

e2e-results/2026-07-30/tb-ro-absent-table-ent-ro-claude/final-message.md

@@ -0,0 +1,17 @@
+Apologies — that last step was unnecessary and shouldn't have used Bash. Here's the result:
+
+```json
+{
+  "found": false,
+  "message": "No table named 'smoke' exists in any database. A database named 'smoke' exists but contains only one table: 'seed_marker'.",
+  "databases_checked": ["_internal", "home", "host_system", "loadcap_probe", "smoke"],
+  "tables_found": {
+    "_internal": [],
+    "home": [],
+    "host_system": ["metrics"],
+    "loadcap_probe": ["loadcap_demo"],
+    "smoke": ["seed_marker"]
+  },
+  "query_executed": null
+}
+```

e2e-results/2026-07-30/tb-ro-core-parity-core-ro-claude/final-message.md

@@ -0,0 +1,15 @@
+Done. Databases: `_internal`, `host_system`, `smoke`. Used `host_system` (non-internal), table `metrics`.
+
+```json
+{
+  "database": "host_system",
+  "table": "metrics",
+  "query": "SELECT * FROM metrics ORDER BY time DESC LIMIT 5",
+  "row_count": 2,
+  "rows": [
+    {"cpu::usage_idle": 45.2, "cpu::usage_system": 9.7, "cpu::usage_user": 45.1, "host": "e2e-host-2", "time": "2026-07-30T04:26:05"},
+    {"cpu::usage_idle": 84.3, "cpu::usage_system": 3.2, "cpu::usage_user": 12.5, "host": "e2e-host-1", "time": "2026-07-30T04:26:05"}
+  ],
+  "duration_ms": 36
+}
+```