@hyperdx/hdx-eval

Eval framework for benchmarking AI agents against observability MCP servers. Generates deterministic synthetic telemetry (traces + logs) with planted anomalies, spawns Claude Code as an SRE agent with access to any configured MCP, records full trajectories, and grades answers using programmatic checks and an LLM-as-judge. The framework is MCP-agnostic — compare any combination of MCPs: HyperDX vs ClickHouse, feature-branch vs main, two different HyperDX instances, or any N-way comparison. MCPs are defined in a config registry and selected at run time.

Prerequisites

yarn dev running on at least one worktree — this handles yarn install, yarn build:common-utils, Docker containers (ClickHouse, MongoDB), and the API server. See agent_docs/development.md for setup.
claude CLI installed (the harness spawns it in streaming JSON mode)
ANTHROPIC_API_KEY or AI_API_KEY in .env.local at the monorepo root (for run and grade commands)
uv installed (only needed if you configure a stdio-type ClickHouse MCP — it launches via uv run --with mcp-clickhouse)

Quick Start

All eval commands derive ClickHouse and API ports from the HDX_DEV_SLOT environment variable. If your dev stack is running in a different worktree, the auto-detected slot won't match. Set HDX_DEV_SLOT to the slot your dev stack is using — you can find this in the banner printed by yarn dev.

# Set the slot for your session (must match the running dev stack)
export HDX_DEV_SLOT=98

# 1. One-time setup: register eval account + create Sources + write config
yarn workspace @hyperdx/hdx-eval dev setup-hyperdx

# 2. Seed a scenario (use --volume-factor 0.1 for faster iteration)
yarn workspace @hyperdx/hdx-eval dev seed error-root-cause --volume-factor 0.1

# 3. Run the eval (all enabled MCPs, 3 runs each)
#    First run saves an anchorTime to eval.config.json automatically.
yarn workspace @hyperdx/hdx-eval dev run error-root-cause

# 4. Re-run later — reuses saved anchor, skips reseed
yarn workspace @hyperdx/hdx-eval dev run error-root-cause

# 5. Grade + report (auto-runs after step 3 by default)
yarn workspace @hyperdx/hdx-eval dev grade <batch>
yarn workspace @hyperdx/hdx-eval dev report <batch>

# 6. Browse results in the web viewer
yarn workspace @hyperdx/hdx-eval viewer

Recommended: Dual-Slot Eval Setup

For reliable A/B comparison of two HyperDX branches (e.g. feature vs main), we recommend reserving two fixed dev slots. This gives you two fully isolated HyperDX stacks (separate ClickHouse, MongoDB, API) that can run simultaneously.

1. Start two dev stacks with fixed slots

# Slot 98: "baseline" — run from your main branch checkout
HDX_DEV_SLOT=98 yarn dev

# Slot 99: "candidate" — run from your feature branch checkout
HDX_DEV_SLOT=99 yarn dev

This gives you:

	Slot 98 (baseline)	Slot 99 (candidate)
API	`http://localhost:30198`	`http://localhost:30199`
App	`http://localhost:30298`	`http://localhost:30299`
ClickHouse	`http://localhost:30598`	`http://localhost:30599`
MCP	`http://localhost:30198/mcp`	`http://localhost:30199/mcp`

2. Run `setup-hyperdx` against each instance

# Setup baseline (slot 98)
yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --api-url http://localhost:30198

# Setup candidate (slot 99)
yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --api-url http://localhost:30199

After setup, eval.config.json will have the hyperdx MCP pointing at the slot it was set up against. You can then manually add the second MCP entry (see MCP Configuration below).

3. Seed both ClickHouse instances

Both instances need the eval data. Use HDX_DEV_SLOT to target each:

HDX_DEV_SLOT=98 yarn workspace @hyperdx/hdx-eval dev seed error-root-cause
HDX_DEV_SLOT=99 yarn workspace @hyperdx/hdx-eval dev seed error-root-cause

4. Run the comparison

yarn workspace @hyperdx/hdx-eval dev run error-root-cause \
  --mcp hyperdx-main,hyperdx-feature \
  --baseline hyperdx-main \
  --runs 5

MCP Configuration

MCPs are defined in eval.config.json under the mcps key. Each entry fully specifies how to connect, which tools to allow/deny, and how to anonymize the MCP identity for fair LLM judging.

Config structure

{
  "mcps": {
    "<name>": {
      "type": "http" | "stdio",
      "url": "...",                    // http only
      "headers": { ... },             // http only
      "command": "...",               // stdio only
      "args": ["..."],               // stdio only
      "env": { ... },                // stdio only
      "toolPattern": "mcp__<name>__*",
      "label": "Display Name",
      "brandTerms": ["Brand", "brand"],
      "deniedTools": ["mcp__<name>__tool_to_hide"],
      "enabled": true
    }
  },
  "scenarios": { ... },
  "hyperdxApi": { ... },
  "clickhouse": { ... }
}

Field reference

Field	Required	Description
`type`	yes	`"http"` for HTTP/SSE transport, `"stdio"` for subprocess
`url`	http	MCP server URL
`headers`	no	HTTP headers (e.g. `Authorization: Bearer ...`)
`command`	stdio	Executable to spawn
`args`	no	Command arguments
`env`	no	Environment variables for the subprocess
`toolPattern`	yes	Glob for allowed tools (e.g. `mcp__myname__*`)
`label`	yes	Human-readable name for reports
`brandTerms`	no	Terms to redact when blinding answers for the LLM judge
`deniedTools`	no	Tool names to deny (reduces tool count for faster schema loading)
`enabled`	no	`false` to exclude from `--mcp all` (default: `true`). Can still be explicitly named with `--mcp name`.

Top-level config sections

Section	Required	Purpose
`mcps`	yes	MCP registry (see above)
`plugins`	no	Claude Code plugin registry (see Plugins)
`scenarios`	no	Per-scenario HyperDX Source IDs (populated by `setup-hyperdx`)
`hyperdxApi`	no	HyperDX API URL + access key (used only by `setup-hyperdx`)
`clickhouse`	no	ClickHouse connection for `seed`, `drop`, `runs-instrument` commands
`anchorTime`	no	Fixed "now" anchor (ISO 8601). Auto-generated and saved on first `run`. See Anchor Time.

Plugins

You can test a Claude Code plugin as a comparison variable — for example a plugin that installs a hook. Define plugins under the top-level plugins key, then select them with --plugin:

{
  "plugins": {
    "myplugin": {
      "label": "My Plugin",
      "url": "https://example.com/my-plugin.zip"
    },
    "dev-plugin": {
      "label": "Dev Plugin",
      "dir": "/absolute/path/to/my-plugin"
    }
  }
}

Field	Required	Notes
`label`	yes	Human-readable label for reports/CLI output
`url`	one of `url`/`dir`	URL to a plugin `.zip` (loaded via `--plugin-url`)
`dir`	one of `url`/`dir`	Local plugin directory (loaded via `--plugin-dir`)

Each entry must set exactly one of url or dir.

--plugin works like --mcp and --model: the variants are the names you pass (comma-separated), and the literal none is the no-plugin variant. When --plugin is omitted the default is the single none variant, so behavior is unchanged.

--plugin myplugin — run only the plugin variant. The plugin doesn't vary, so the column key is just <mcp>.
--plugin none,myplugin — run both the no-plugin baseline and the plugin variant (column keys <mcp>/none and <mcp>/myplugin), so the report shows the plugin variant as a challenger with per-metric deltas vs. the no-plugin baseline. The baseline defaults to the first variant (none here); override with --baseline.
--plugin none,a,b — compare several plugins against no-plugin at once.

Each named plugin is loaded into the isolated agent session via --plugin-url (for url) or --plugin-dir (for dir).

Example: HTTP MCP (HyperDX)

{
  "type": "http",
  "url": "http://localhost:30198/mcp",
  "headers": { "Authorization": "Bearer <access-key>" },
  "toolPattern": "mcp__hyperdx__*",
  "label": "HyperDX",
  "brandTerms": ["HyperDX", "hyperdx"],
  "deniedTools": [
    "mcp__hyperdx__hyperdx_delete_dashboard",
    "mcp__hyperdx__hyperdx_get_dashboard"
  ]
}

Example: stdio MCP (raw ClickHouse)

{
  "type": "stdio",
  "command": "uv",
  "args": ["run", "--with", "mcp-clickhouse==0.3.0", "--python", "3.10", "mcp-clickhouse"],
  "env": {
    "CLICKHOUSE_HOST": "localhost",
    "CLICKHOUSE_PORT": "30598",
    "CLICKHOUSE_USER": "default",
    "CLICKHOUSE_PASSWORD": ""
  },
  "toolPattern": "mcp__clickhouse__*",
  "label": "ClickHouse MCP",
  "brandTerms": ["ClickHouse MCP", "clickhouse"]
}

Scenarios

Each scenario writes into its own ClickHouse tables (eval_<scenario>_otel_traces / eval_<scenario>_otel_logs) whose schema mirrors HyperDX's otel_traces / otel_logs exactly.

Scenario	Agent Prompt	What's Planted
`error-root-cause`	"Checkout requests are failing..."	`payment-service` DB timeout cascading into `checkout-api` 5xx. 6M+ spans, 12M logs, 5 distractors.
`deploy-regression`	"Support has been fielding a growing trickle of 'something went wrong at checkout' reports..."	`checkout-api` staged rollout on 3/6 pods → ~7% checkout 500s. Root cause is in traces/logs via `k8s.pod.name` cross-tab; metrics only corroborate. 6 distractors.
`latency-spike`	"p99 latency on api-server has jumped..."	`/api/orders/search` p99 spike for enterprise tenants. 12M+ spans, 5 regions, 5 distractors.
`metric-saturation`	"Shoppers are seeing slow/missing recommendations; frontend-proxy logs upstream 503s..." (blinded — culprit service not named)	`recommendation-service` JVM heap leak → GC-pause tail → latency shift → pod-restart cadence. Root cause is metrics-only (per-pod × per-pool heap gauges, all five metric types), with 5 distractors incl. a healthy JVM twin.
`noisy-signals`	"We want to cut log ingest cost..."	~16M logs across composite cells. Each noisy cell has a load-bearing pattern mixed in.
`segmented-regression`	"API error rate has gone up..."	Enterprise x cache-miss errors at ~12%. Single-axis aggregates dilute the signal.
`service-health-check`	"Generate a service health report..."	Peace-time report — no incident, but 4 novel signals to call out.
`dashboard-build`	"We need dashboards to monitor our microservices..."	3 user-facing + 4 distractor services. Dashboard creation via MCP tools, cross-dashboard drill-down, messy severity, latency red herring, impossible metric requests.

Use --volume-factor to scale background row counts for faster iteration. Planted anomaly counts stay fixed.

# Full volume (default) — realistic needle-in-haystack difficulty
yarn workspace @hyperdx/hdx-eval dev seed error-root-cause

# 10% volume — much faster, still structurally valid
yarn workspace @hyperdx/hdx-eval dev seed error-root-cause --volume-factor 0.1

CLI Reference

All commands are run via yarn workspace @hyperdx/hdx-eval dev <command>.

Setup

dev setup-hyperdx                           # one-time: register + create Sources
dev setup-hyperdx --api-url http://...      # explicit API URL
dev setup-hyperdx --check                   # validate existing config
dev setup-hyperdx --reset-sources           # recreate all Sources

Data Management

dev list                                    # list scenarios + tables
dev seed <scenario>                         # seed with defaults (seed=42, now=Date.now())
dev seed <scenario> --seed 7                # custom PRNG seed
dev seed <scenario> --now 2026-05-08T15:00:00Z  # fixed anchor time
dev seed <scenario> --volume-factor 0.1     # 10% background volume
dev seed <scenario> --ch-url http://localhost:30599  # target specific CH
dev drop <scenario>                         # drop scenario tables

Running Evals

dev run <scenario>                          # all enabled MCPs, 3 runs each
dev run <scenario> --mcp hyperdx            # single MCP
dev run <scenario> --mcp hyperdx-main,hyperdx-feature  # compare two
dev run <scenario> --mcp all                # all enabled MCPs (default)
dev run <scenario> --baseline hyperdx-main  # set baseline for delta reports
dev run <scenario> --plugin none,myplugin   # compare no-plugin baseline vs plugin variant
dev run <scenario> --runs 5                 # 5 runs per cell
dev run <scenario> --model claude-opus-4-6
dev run <scenario> --max-turns 20           # default: 15
dev run <scenario> --concurrency 4          # parallel cells
dev run <scenario> --prompt-variant hypothesis  # hypothesis-first variant
dev run <scenario> --reseed                 # re-seed data before running
dev run <scenario> --anchor-time 2026-05-08T15:00:00Z  # override + save anchor
dev run <scenario> --live                   # wall-clock now, no anchor, forces reseed
dev run <scenario> --no-grade               # skip auto-grading
dev run <scenario> --no-judge               # programmatic checks only (faster)

Inspecting Results

dev runs-list                               # list recorded batches
dev runs-show <path>                        # summary of a run
dev runs-show <path> --queries              # every tool call + SQL
dev runs-show <path> --final-answer         # print the agent's answer
dev runs-instrument <batch>                 # enrich with ClickHouse query_log timings

Grading & Reporting

dev grade <batch>                            # programmatic + LLM-as-judge
dev grade <batch> --judge-model openai:gpt-4o        # grade with a different provider
dev grade <batch> --no-judge                 # programmatic checks only
dev grade <batch> --rerun-judge              # force re-grade with the current judge
dev report <batch>                           # render _summary.md + _summary.json
dev report <batch> --baseline hyperdx-main   # override baseline for deltas

Independent grader (different provider/model)

The LLM-as-judge can run on a different provider/model than the run model (Anthropic run graded by OpenAI, and vice-versa), which reduces same-model bias. Pass --judge-model as a provider:model spec — supported providers anthropic and openai; a bare model name defaults to anthropic:

dev grade <batch> --judge-model openai:gpt-4o
dev grade <batch> --judge-model claude-opus-4-7   # == anthropic:claude-opus-4-7

Precedence: --judge-model flag → eval.config.json grading.judgeModel → built-in default (anthropic:claude-opus-4-7):

{ "grading": { "judgeModel": "openai:gpt-4o" } }

Grades are cached per run; re-grading a batch with a different judge re-runs that judge automatically (a cached grade from another judge model is treated as stale). Use --rerun-judge to force a refresh with the same judge.

Grader credentials come from the environment, mirroring packages/api: OPENAI_API_KEY / ANTHROPIC_API_KEY per provider, AI_API_KEY as a fallback, AI_BASE_URL for a custom endpoint (Azure AI, LiteLLM proxy), and AI_REQUEST_HEADERS (OpenAI) as a JSON object. The provider-specific key wins over AI_API_KEY, so the runner's Anthropic key can live in AI_API_KEY while an OpenAI grader uses OPENAI_API_KEY without the two colliding.

Viewer

yarn workspace @hyperdx/hdx-eval viewer    # web UI at http://localhost:5176

The viewer shows a comparison dashboard when you select a batch:

Top-line verdict table — combined scores per MCP with delta columns
Per-scenario breakdown — metrics, judge criterion heatmap, programmatic check pass rates
Side-by-side runs — click any run to drill into the full trajectory

Scoring

Combined score = 0.4 * programmatic + 0.6 * judge - toolErrorPenalty

Programmatic (40%) — regex-based keyword checks against the final answer, defined per-scenario in ground-truth.json. Includes both positive checks (must find the right thing) and negative checks (must not blame a distractor).
LLM-as-judge (60%) — scenario-specific, multi-criteria rubric scored 0-5 by an LLM judge (configurable — see Independent grader). The judge prompt defines explicit 0-5 scoring anchors (0 = wrong/absent … 5 = fully correct) to keep scores calibrated across judge models, and answers are blinded — MCP tool names and brand terms are redacted so the judge can't tell which MCP produced the answer.

Judge scores are only comparable within the same judge model (anchors reduce but don't erase per-model scale differences). Hold the judge constant across runs you compare directly.
Tool error penalty — up to 20% deducted for high tool-call error rates.

Adoption (tool use) is reported alongside the outcome score for scenarios with an adoption rubric block. Each check declares the scenario's target metric names/keys; a tool call counts when its input args name one of them (separator-tolerant: jvm.gc.pause ≡ jvm_gc_pause), regardless of which tool was called — a dedicated metric tool and raw SQL against the metrics tables score identically, so both arms are graded by the same rule. Tool names, tool outputs, and the prompt never count. Adoption is intentionally excluded from the combined score: measuring tool usage must not inflate outcome quality. metric-saturation (metrics load-bearing) should show ≈100% adoption on its four core checks; the interesting readout is deploy-regression (metrics merely corroborating), where adoption measures whether the agent reaches for metrics it doesn't strictly need.

metric-saturation includes two opportunistic checks (discovered_summary_metric, queried_summary_via_sql) on the db.client.operation.duration Summary distractor, which is discoverable via MCP but only queryable through raw clickstack_sql. They measure whether the agent follows that discovery → SQL path; since a strong run may justifiably skip the distractor after finding the JVM leak, sub-100% scores are expected.

Baseline + Challengers

Reports use a baseline + challengers model. One column is designated as the baseline (default: the first mcp/model/plugin set, or set explicitly with --baseline <column-key>). All other columns show deltas relative to it:

| Metric         | hyperdx-main | hyperdx-feature | Δ (hyperdx-feature) |
|----------------|-------------|-----------------|---------------------|
| Combined score | 85%         | 91%             | +6%                 |

Determinism

All randomness flows through a mulberry32(seed) PRNG. Same (seed, now) produces byte-identical row content. This makes side-by-side comparison meaningful — both MCPs query the exact same data.

Anchor Time

The anchor time is the "now" reference for both seeded data timestamps and the agent's system prompt. It is persisted in eval.config.json under anchorTime so that subsequent runs automatically reuse the same value.

First run: if anchorTime is not set in the config, the current wall-clock time is saved and used. If the scenario tables are empty or missing, data is auto-seeded — no separate seed step required.
Subsequent runs: the saved anchorTime is read from config. Data is verified to exist before running; if it's missing (e.g. after a drop), it's auto-seeded again.
--anchor-time <iso>: override the saved value with a specific timestamp. The new value is saved to config for future runs. Pair with --reseed if the data needs to be regenerated to match.
--live: ignore the saved anchor and use wall-clock time. The agent does NOT receive a FIXED CURRENT TIME system prompt block. Implies --reseed since the data must match the current time.
--reseed: force re-seed even when data already exists (e.g. after changing the seed value or anchor time).

The anchor is intentionally allowed to age. It is not auto-refreshed when wall-clock time advances, and stale data does not trigger an automatic reseed. This keeps seeded data stable across runs (important for CI, where a fresh reseed is much slower than reusing cached seed data).

Note on clickstack_describe_source: that tool samples its value-related fields (lowCardinalityValues, mapAttributeValues, mapAttributeKeys) over a fixed last-24h-of-wall-clock window, independent of the anchor. Once seeded data ages past that window those fields come back empty/partial. Rather than refreshing the anchor and reseeding to keep them populated, the system prompt instructs the agent to treat those samples as unreliable and to discover real values via anchored queries instead (see SAMPLING_CAVEAT_BLOCK in src/harness/systemPrompt.ts). The columns schema from describe_source is unaffected and remains reliable.

Tests

yarn workspace @hyperdx/hdx-eval ci:unit   # all unit tests
yarn workspace @hyperdx/hdx-eval dev:unit  # watch mode

Unit tests verify PRNG determinism, planted anomaly invariants, grading pipeline, stream parsing, prompt construction, blinding, and report aggregation. No ClickHouse needed.

Navigation Menu

Repository navigation

README.md

README.md

@hyperdx/hdx-eval

Prerequisites

Quick Start

Recommended: Dual-Slot Eval Setup

1. Start two dev stacks with fixed slots

2. Run `setup-hyperdx` against each instance

3. Seed both ClickHouse instances

4. Run the comparison

MCP Configuration

Config structure

Field reference

Top-level config sections

Plugins

Example: HTTP MCP (HyperDX)

Example: stdio MCP (raw ClickHouse)

Scenarios

CLI Reference

Setup

Data Management

Running Evals

Inspecting Results

Grading & Reporting

Independent grader (different provider/model)

Viewer

Scoring

Baseline + Challengers

Determinism

Anchor Time

Tests

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

@hyperdx/hdx-eval

Prerequisites

Quick Start

Recommended: Dual-Slot Eval Setup

1. Start two dev stacks with fixed slots

2. Run setup-hyperdx against each instance

3. Seed both ClickHouse instances

4. Run the comparison

MCP Configuration

Config structure

Field reference

Top-level config sections

Plugins

Example: HTTP MCP (HyperDX)

Example: stdio MCP (raw ClickHouse)

Scenarios

CLI Reference

Setup

Data Management

Running Evals

Inspecting Results

Grading & Reporting

Independent grader (different provider/model)

Viewer

Scoring

Baseline + Challengers

Determinism

Anchor Time

Tests

2. Run `setup-hyperdx` against each instance