🕵️ poirot-agent

A coding agent that gets paged instead of a human — and investigates the incident under a read-only lock.

Poirot is Claude Code running headless on shared AWS compute. When a CloudWatch alarm fires on a log-error spike, Poirot is dispatched automatically, investigates the incident read-only, and writes a grounded root-cause report to your inbox. It diagnoses; it never touches production.

It's the "headless coding agent on shared cloud compute that a whole team can trigger" pattern — inspired by headless-claude-on-aws — pointed at incident response and written in TypeScript (AWS CDK + a thin runner that drives Claude Code).

The idea in one picture

   CloudWatch alarm  (error-spike metric filter)
          │  alarm action
          ▼
        SNS · AlarmTopic ──(undeliverable)──▶  SQS · TriggerDLQ ──▶ depth alarm ──▶ 📧
          │
          ▼
    Trigger Lambda  ── dedup + circuit breaker against recent builds ──
          │  StartBuild (unless suppressed — an ack/suppression notice
          │   goes to ReportsTopic either way)
          ▼
      CodeBuild · "poirot-investigator"
          │
          │  installs Claude Code, runs the TS runner
          ▼
 claude -p  (headless, stream-json)
          │  Bash → AWS CLI
          ▼  ── under the READ-ONLY investigator role ──
CloudWatch Logs Insights · metrics · deploy history
          │
          ▼
Root-cause report ──▶ build log + SNS · ReportsTopic ──▶ 📧

No human in the loop until the report lands. An alarm becomes an investigation; an investigation becomes a report — and if it doesn't, the DLQ, the circuit breaker, and Poirot's own self-monitoring alarms all report back to the same ReportsTopic so that failure isn't silent either.

Why this is interesting (the talk track)

Three design choices do the heavy lifting — each is a slide on its own:

1. The agent is structurally incapable of causing harm

Poirot runs under a dedicated read-only IAM role, separate from the role that launches it. Even if the model is confused, prompt-injected by a malicious log line, or simply wrong, it cannot create, modify, restart, scale, or delete anything — the credentials don't allow it. Safety is enforced by IAM, not by asking the model nicely.

build role        →  can ONLY: assume the investigator role, publish a report,
                      read the Claude token            (least privilege)
investigator role →  ReadOnlyAccess — what Claude's AWS CLI calls actually run as
                      ...minus an explicit DENY on secrets, SSM params, KMS
                      decrypt, S3 object reads, and DynamoDB data

Read-only isn't enough on its own: Poirot reads untrusted log content (a prompt-injection surface) and then publishes a report, so a crafted log line must not be able to talk it into reading a secret and leaking it. An explicit deny on the high-value data reads closes that door — it can investigate infrastructure (logs, metrics, deploys, config) but cannot read your data.

This is the crux of trusting an autonomous agent in production: you don't trust the agent, you trust the blast-radius wall around it.

2. It bills against a subscription, not per token

Claude Code authenticates with a Claude Pro/Max subscription token (CLAUDE_CODE_OAUTH_TOKEN from claude setup-token). Investigations draw down your plan at a flat cost — no per-incident API metering, no surprise bill when a noisy night fires fifty alarms.

3. Incident response is a near-perfect agent task

It's read-heavy, bounded, and repetitive — exactly what burns out on-call engineers and exactly where an agent shines: pull the error signatures, correlate with the last deploy, size the blast radius, write it up. Poirot does the first 30 minutes of every investigation so a human starts from a hypothesis instead of a blank terminal.

What a report looks like

Poirot always finishes with one self-contained report (structure enforced by system-prompt.md):

## Incident summary
checkout-api 5xx rate jumped from ~0.1% to 18% at 14:02 UTC and is ongoing.

## Root cause
Deploy `d-AB12CD` (14:01 UTC) shipped a config change that points the service at
a connection pool of 5; under normal traffic it exhausts immediately, surfacing
as "FATAL: remaining connection slots are reserved".

## Evidence
- Logs Insights: 9,412 × "remaining connection slots are reserved", first seen 14:02:11 — zero before 14:02.
- CodeDeploy: deployment d-AB12CD completed 14:01:48, one minute before onset.
- CloudWatch: DatabaseConnections flatlined at the new ceiling from 14:02.

## Blast radius
All checkout traffic in us-east-1; ~18% of requests failing. Read paths unaffected.

## Confidence
high — the deploy timestamp, the new error signature, and the connection ceiling all line up.

## Recommended next steps
1. Roll back d-AB12CD or raise the pool size.
2. Add a pre-deploy check on the pool-size config.

How it works

Stage	What happens
Trigger	A CloudWatch alarm (e.g. an error-spike metric filter) fires its action to the `AlarmTopic`. The trigger Lambda parses it, checks the last ~30 builds for a duplicate or a chronically firing service, and either suppresses the dispatch (with a notice to `ReportsTopic`) or calls `StartBuild` with `LOG_GROUPS`/`SERVICE`/`WINDOW_START`/`WINDOW_END` derived from the alarm.
Runtime	CodeBuild installs Claude Code and runs the TS runner, which builds the prompt, spawns `claude -p --output-format stream-json`, and parses the event stream for Poirot's final report.
Tools	Claude Code's Bash tool running the AWS CLI — no MCP server, no custom SDK tools. Logs Insights, metrics, and deploy history are all just CLI calls.
Output	The report is printed to the build log and published to the `ReportsTopic` (subscribe email, Slack, PagerDuty, …).

The detective's method

Poirot works the case in a fixed order — establish the facts, read the actual error lines, correlate with recent deploys/changes, size the blast radius, then form and try to disprove a hypothesis before committing. Every claim is tied to a log line, metric, or deploy event it actually retrieved. The full method and hard rules live in system-prompt.md.

Resilience: dedup, circuit breaker, and self-monitoring

A single incident can trip several alarms, and a flapping alarm can re-fire the same one repeatedly — the trigger Lambda absorbs both without a database, by querying CodeBuild's own recent build history:

Dedup. Keyed on (SERVICE, METRIC_NAME) — pulled from the alarm's dimensions and Trigger.MetricName — with a 10-minute window. Two distinct metrics on the same service (Errors vs Throttles) are not treated as duplicates of each other, since they're usually separate investigations.
Circuit breaker. If a service has already been investigated 3+ times in the last hour, further dispatches for it are suppressed — a chronically firing alarm stops burning Claude subscription turns on repeat diagnoses of the same root cause.
Investigator gets tighter windows. WINDOW_START/WINDOW_END are derived from the alarm's own Trigger.Period × EvaluationPeriods when present (a 2-of-3 × 5-min alarm → a 15-minute window) instead of a flat 1 hour, and LOG_GROUPS falls back to /aws/lambda/<FunctionName> when the alarm has no explicit log-group dimension.
Nothing fails silently. Every dispatch, suppression, and circuit-open decision publishes a short notice to ReportsTopic, so operators see Poirot's state instead of guessing whether an alarm fired at all. A DLQ on the AlarmTopic subscription catches anything StartBuild itself can't swallow (e.g. a concurrent-build limit), with a depth alarm watching it. And a handful of self-monitoring CloudWatch alarms — trigger Lambda errors/ throttles, DLQ depth, investigator build failures — publish to ReportsTopic too, so a broken watcher doesn't go unnoticed without feedback-looping back into Poirot itself.

Repository layout

Path	What
`system-prompt.md`	Poirot's persona, investigation method, and hard rules
`src/prompt.ts`	Reads the investigation context from env; builds the system + case prompts
`src/claude.ts`	Spawns `claude -p`, parses the stream-json events into a report
`src/investigate.ts`	Entrypoint: run the investigation, print + publish the report
`buildspec.yml`	CodeBuild: install Claude Code, run the TS runner
`infra/lib/poirot-stack.ts`	CDK: CodeBuild, dual IAM roles, SNS topics, DLQ, self-monitoring alarms, trigger Lambda, GitHub OIDC deploy role, example alarm
`infra/lambda/trigger.ts`	SNS alarm → `decideDispatch()` (dedup + circuit breaker) → `StartBuild` with per-incident env overrides
`test/unit.test.ts`	Dependency-free `node:test` unit tests (`npm test`)
`.github/workflows/ci.yml`	On every PR: typecheck, test, `cdk synth`
`.github/workflows/deploy.yml`	On push to `main`: typecheck, test, then `cdk deploy` via OIDC (no long-lived AWS keys in GitHub)
`demo/`	Live-demo kit — see Live demo kit below

Deploy

Prerequisites: an AWS account (CDK-bootstrapped), a Claude Pro/Max subscription, and a one-time GitHub source credential so CodeBuild can clone:

aws codebuild import-source-credentials \
  --server-type GITHUB --auth-type PERSONAL_ACCESS_TOKEN --token "$GITHUB_PAT"

npm install
npm run deploy            # cdk deploy PoirotStack

This first deploy has to run locally — it's what creates the GitHub OIDC provider and deploy role in the first place, so GitHub Actions has nothing to assume before it exists. Once it's up, hand deploys off to CI: grab GitHubDeployRoleArn from the stack outputs, set it as the repo secret AWS_DEPLOY_ROLE_ARN (and optionally the repo variable AWS_REGION), and every push to main that passes .github/workflows/ci.yml's checks auto-deploys via .github/workflows/deploy.yml — OIDC, so no long-lived AWS keys live in GitHub. PRs only run the CI checks; they never deploy.

After deploy, mint a subscription token, store it, and subscribe to reports:

claude setup-token        # long-lived token tied to your Pro/Max plan

aws secretsmanager put-secret-value \
  --secret-id poirot-agent/claude-token --secret-string "$CLAUDE_CODE_OAUTH_TOKEN"

aws sns subscribe --protocol email \
  --topic-arn "$(aws cloudformation describe-stacks --stack-name PoirotStack \
    --query "Stacks[0].Outputs[?OutputKey=='ReportsTopicArn'].OutputValue" --output text)" \
  --notification-endpoint you@example.com

cdk.json pins claudeModel to a specific Sonnet snapshot by default (cheaper and more predictable per investigation than floating to whatever "default" means at deploy time); override it, or wire the example error-spike alarm to one of your log groups:

npm run deploy -- -c claudeModel=claude-sonnet-4-6
npm run deploy -- -c targetLogGroupName=/aws/lambda/my-service

Run an investigation

Manually:

aws codebuild start-build --project-name poirot-investigator \
  --environment-variables-override \
    name=TRIGGER,value="checkout 5xx spike" \
    name=LOG_GROUPS,value="/aws/ecs/checkout,/aws/lambda/checkout-api" \
    name=SERVICE,value="checkout"

Automatically: point any CloudWatch alarm's action at the AlarmTopic ARN from the stack outputs. The trigger Lambda does the rest.

Per-investigation inputs

Var	Required	Meaning
`TRIGGER`	✅	Alarm name / incident title
`LOG_GROUPS`		Comma-separated candidate log groups. When the trigger Lambda builds this from an alarm, it falls back to `/aws/lambda/<FunctionName>` if the alarm has no explicit log-group dimension.
`SERVICE`		Service/app name to narrow the search — also the dedup/circuit-breaker key
`METRIC_NAME`		The alarm's metric (e.g. `Errors`, `Throttles`) — the other half of the dedup key alongside `SERVICE`
`WINDOW_START` / `WINDOW_END`		ISO-8601 window. When dispatched from an alarm, derived from `Trigger.Period × EvaluationPeriods`; otherwise defaults to the last hour.
`RAW_PAYLOAD`		Raw alarm/incident JSON, passed through for context
`CLAUDE_MODEL`		Model override (else `cdk.json`'s pinned default)
`CLAUDE_MAX_TURNS`		Cap on agent turns (default 40)

Stack-level vars (INVESTIGATOR_ROLE_ARN, REPORT_SNS_TOPIC_ARN) are set by CDK; the Claude token arrives from Secrets Manager as CLAUDE_CODE_OAUTH_TOKEN. The AlarmTopic ARN is also exported to SSM as /poirot/alarm-topic-arn, so other stacks can point their alarms at it without a CDK cross-stack dependency.

Live demo kit

Showing this at a conference? demo/ has a script that pumps a realistic error burst into a scratch log group so the whole pipeline — alarm fires, Poirot investigates, report lands — happens live on stage, plus a canned fallback report for when wifi doesn't cooperate mid-talk. See demo/README.md.

Token lifecycle

claude setup-token is the long-lived, headless credential — the same path Anthropic's own claude-code-action uses. The token is tied to your plan and lasts on the order of months (up to ~a year), so day to day there's nothing to manage. When it eventually expires you'll see an auth error in the build log; rotate it in place, no redeploy:

claude setup-token
aws secretsmanager put-secret-value \
  --secret-id poirot-agent/claude-token --secret-string "$CLAUDE_CODE_OAUTH_TOKEN"

A zero-touch variant is possible — persist Claude Code's auto-refreshed OAuth credentials back to Secrets Manager after each build. It needs secretsmanager:PutSecretValue on the build role and serialized builds (concurrentBuildLimit: 1) to avoid refresh-token races, so it's intentionally left out in favour of the simpler rotate-when-it-expires approach.

Design decisions (FAQ / Q&A)

Why a separate read-only role instead of just trusting the prompt? Because prompts are not a security boundary. A malicious string in a log line could try to talk the agent into running a mutating command. IAM is the boundary; the agent literally lacks the permission, so it doesn't matter what it's told.

Why subscription billing rather than an API key? An API key bills per token — fine, but unpredictable when alarm storms hit. A subscription token is flat cost and the same credential Claude Code is designed to run headless with. Swapping to an API key is a one-line secret change if you'd rather meter per token.

Why the AWS CLI over an AWS MCP server? Fewer moving parts. The CLI is already authoritative, every read maps to an IAM action the investigator role can be scoped to, and there's nothing extra to install or keep in sync. MCP is a clean upgrade path if you want richer tooling.

Why CodeBuild instead of Lambda? Investigations are long, bursty, and need a real shell with the AWS CLI and Node toolchain. CodeBuild gives that with no idle cost and natural concurrency.

What stops it running forever / racking up cost? CLAUDE_MAX_TURNS (default 40) bounds the agent, and CodeBuild's own timeout bounds the build. The circuit breaker bounds it further at the alarm level — no more than 3 investigations per service per hour, regardless of how often the alarm fires.

Why key dedup on (SERVICE, METRIC_NAME) instead of just the alarm name? The same alarm firing twice is an obvious duplicate, but so is Errors and Throttles both firing for checkout-api within minutes of each other — same service, likely same root cause, and investigating both burns two turns' worth of subscription budget for one answer. Keying on the pair catches that without collapsing genuinely distinct alarms (Errors on checkout-api and Errors on payments-api) into one.

Why query CodeBuild's own build history for dedup instead of a database? One fewer stateful resource to provision, secure, and pay for. CodeBuild already remembers what ran, when, and with what environment variables — that's exactly the state dedup needs, and it's already there.

Why a DLQ and self-monitoring alarms — isn't read-only + dedup enough? Those handle the agent behaving unexpectedly; the DLQ and self-monitoring alarms handle the harness behaving unexpectedly — a throttled Lambda, a StartBuild that throws, a chronically failing build. Without them, "the alarm fired but nothing happened" is silent. Both failure classes need watching for an unattended system to be trustworthy.

Why OIDC for the GitHub Actions deploy role instead of AWS access keys? No long-lived credentials sitting in GitHub secrets waiting to leak. The role trusts only token.actions.githubusercontent.com for this repo's main branch, and a session lasts at most an hour.

Local development

npm install
npm run typecheck
npm test
npx cdk synth --quiet   # sanity-checks the stack without deploying
# Drive one investigation locally — needs Claude Code on PATH, an authed Claude
# session (or CLAUDE_CODE_OAUTH_TOKEN), and AWS credentials:
TRIGGER="local test" LOG_GROUPS="/aws/lambda/foo" npm run investigate

Navigation Menu

Repository files navigation

🕵️ poirot-agent

The idea in one picture

Why this is interesting (the talk track)

1. The agent is structurally incapable of causing harm

2. It bills against a subscription, not per token

3. Incident response is a near-perfect agent task

What a report looks like

How it works

The detective's method

Resilience: dedup, circuit breaker, and self-monitoring

Repository layout

Deploy

Run an investigation

Per-investigation inputs

Live demo kit

Token lifecycle

Design decisions (FAQ / Q&A)

Local development

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
demo		demo
infra		infra
src		src
test		test
.gitignore		.gitignore
README.md		README.md
buildspec.yml		buildspec.yml
cdk.json		cdk.json
package-lock.json		package-lock.json
package.json		package.json
system-prompt.md		system-prompt.md
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation

🕵️ poirot-agent

The idea in one picture

Why this is interesting (the talk track)

1. The agent is structurally incapable of causing harm

2. It bills against a subscription, not per token

3. Incident response is a near-perfect agent task

What a report looks like

How it works

The detective's method

Resilience: dedup, circuit breaker, and self-monitoring

Repository layout

Deploy

Run an investigation

Per-investigation inputs

Live demo kit

Token lifecycle

Design decisions (FAQ / Q&A)

Local development

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages