Skip to content

fix(bb-agent): #939 rebased + review fixes - conversation ownership checks - #1016

Draft
belaboe wants to merge 8 commits into
mainfrom
review/939-fixes
Draft

belaboe wants to merge 8 commits into
mainfrom
review/939-fixes

Conversation

@belaboe

@belaboe belaboe commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

#939 rebased + review fixes.

This draft carries #939's commit (d58e309, rebased onto current main) plus the fixes for the review on #939 (seven more commits). It targets main instead of #939's branch: #939's branch is behind main, so a PR stacked on it would also show every main commit since then. Merge this one instead of #939, or cherry-pick the seven follow-up commits onto #939's branch after rebasing it.

What changes (conversation ownership checks in @aws-blocks/bb-agent)

  • Ownership read is strongly consistent (review: blocking, agent.ts ownsConversation). New DistributedTable.get(key, { consistentRead: true }) (GetOptions, maps to ConsistentRead: true on AWS; the mock is always consistent; re-exported from @aws-blocks/blocks as DTGetOptions). The ownership check uses it, so stream() right after createConversationId() is not rejected on AWS.
  • getPendingInterrupts() without a userId throws AgentErrors.InvalidUsage naming the signature (review: blocking).
  • getConversation(id, userId, options?) is owner-scoped (review open question 2).
  • Realtime chunk channels are user-scoped with persistence (review open question 3): a turn's chunks are published on u/<encoded userId>/<channelId>, and getChannel(channelId, userId) resolves the same key, so a caller-chosen channelId can no longer publish onto, or subscribe to, another user's live channel.
  • A turn without a conversationId gets its own AgentCore session on AWS. Found in this follow-up's own review: the runtime session id was the caller-chosen channelId and the AgentCore entry used it as the conversation, which bypassed the ownership check. The entry now takes conversationId from the request only, as the mock already behaved, and each such turn gets a fresh turn-<uuid> session.
  • Background turns never report an error on the bare caller-chosen channel and no longer leave unhandled rejections.
  • inferenceOnly agents keep no owner; their channels stay unscoped. The README now says that a conversationId passed to one is a shared session key.

Breaking changes (0.x minor, flagged for maintainer review)

The original reviewer recommended each of these:

  • getPendingInterrupts(conversationId, userId): positional userId, as in fix(bb-agent): check conversation ownership in stream, resume, and getPendingInterrupts #939 (the reviewer's suggestion block keeps it positional; it matches deleteConversation/listConversations/createConversationId, and TypeScript callers get a compile error that points at the fix).
  • getConversation(id, userId, options?): new required second argument, before { limit }.
  • getChannel(channelId, userId): new required second argument, and the method is now async. inferenceOnly agents don't use it.
  • AWS only: a deployed agent that reused a channelId without a conversationId to keep memory across turns loses that memory; pass a conversationId.
  • With persistence, the realtime channel key changes, so chat sessions that were open during the deploy need a reload.

Changesets: @aws-blocks/bb-agent minor, @aws-blocks/bb-distributed-table minor (consistentRead), @aws-blocks/blocks minor (re-exports Agent and DTGetOptions). At release, changesets moves bb-agent's internal range on bb-distributed-table to the new minor. No third-party dependency is added or upgraded.

Review comments on #939

Comment Change
Blocking: eventually consistent ownership read consistentRead option, used by ownsConversation
Blocking: old one-argument getPendingInterrupts InvalidUsage guard + unit test
Minor: inferenceOnly exemption untested unit test (stream() with a conversationId on an inferenceOnly agent completes)
Minor: End-to-End example lacks getPendingInterrupts/resume backend handlers added and wired in the frontend snippet
Nit: getCurrentUser then user.userId requireAuth in all three server snippets
Breaking-change options positional userId kept, guard added
OQ1 consistent read as above
OQ2 getConversation scoping owner-scoped here
OQ3 caller-chosen channelId fixed here (user-scoped channels) rather than a follow-up
Rebase on main done
No AWS E2E for cross-user rejection test-apps/comprehensive/test/agent.test.ts "Cross-user conversation ownership": user B stream/resume/read/push/listen on user A's conversation, plus a conversation-less turn onto A's id

Tests

  • Red first: the new unit and e2e tests failed against fix(bb-agent): check conversation ownership in stream, resume, and getPendingInterrupts #939's head (read, push and listen; the AgentCore session cases failed before their fix), and they pass now. Twelve mutants of the new guards and orderings each fail at least one test.
  • Local lanes on the final head (AL2023 container, Node 22): npm run build, npm run lint:deps, npm test, npm run test:e2e:local, npm run lint all exit 0.
  • CI on f4c9d30: Build, Unit Tests, E2E Local pass; E2E Sandbox passes (401 tests, including "Cross-user conversation ownership" against AWS); CodeQL passes (an earlier head was flagged for hashing the user id into the AgentCore session id; that hash is gone).
  • E2E Production fails at synth for a reason unrelated to this diff: the test app's derived bucket name bb-test-prod-pr-1016-1-runner-...-test-app-preset-balanced-sn is 64 characters, one over S3's limit, because this PR number has four digits. The bucket and agent ids are main's.

Residuals

  • bb-realtime client: a second subscription to a channel that is already open on the same connection never settles its established promise (the client does not re-send subscribe). Seen while writing the single-client e2e; not changed here.
  • A channelId near Realtime's 1024-byte path limit now fails a little sooner because of the u/<user>/ prefix. The turn reports it in the logs instead of crashing the process.

belaboe and others added 7 commits October 23, 2026 13:57
…tPendingInterrupts

stream() and resume() accepted any conversationId with a caller-supplied
userId and never checked that the conversation belonged to that user, so
a non-owner could add turns to another user's conversation or approve
(and trust) their pending tool call. Both now reject with
ConversationNotFound unless the { userId, conversationId } record exists,
the same check deleteConversation() already used.

getPendingInterrupts() now takes the caller's userId and returns [] for a
conversation that user doesn't own.

The README End-to-End example took userId from the request body; it now
derives it from the authenticated session.
…-scoped reads and channels

Follow-up to the conversation ownership checks, applying the review:

- The ownership read is now strongly consistent. DistributedTable.get()
  takes a new GetOptions { consistentRead } (maps to ConsistentRead on
  AWS; the mock is always consistent), and ownsConversation() uses it,
  so stream() right after createConversationId() is not rejected on AWS.
- getPendingInterrupts() without a userId throws InvalidUsage naming the
  new signature instead of returning [] on the mock and an opaque
  marshalling error on AWS.
- getConversation(id, userId, options?) is owner-scoped: [] for a
  conversation the caller does not own, InvalidUsage without a userId.
- With persistence, a turn's chunks are published on a channel scoped to
  the user who started it, and getChannel(channelId, userId) resolves the
  same key, so a caller-chosen channelId can no longer publish onto, or
  subscribe to, another user's live channel. inferenceOnly agents keep
  unscoped channels and an optional userId.
- Tests: unit tests for each of the above plus the inferenceOnly
  exemption, and a cross-user e2e in test-apps/comprehensive (user B
  stream/resume/read/push/listen on user A's conversation).
- Docs: README Authorization section and End-to-End backend (adds the
  getPendingInterrupts and resume handlers, uses requireAuth), DESIGN.md
  for both packages, changesets, API reports.
…tion sessions

On AWS a turn without a conversationId used the caller-chosen channelId
as its AgentCore runtimeSessionId, and the AgentCore entry fell back to
that session id as the conversationId. stream({ channelId: <another
user's conversation id> }) without a conversationId therefore skipped
the ownership check and ran inside that conversation's session. The
entry now takes conversationId from the request only (toTurnPayload),
and such turns get a session derived from their user and channel, as
the local mock already behaved.

Also from review:
- getChannel(channelId, userId) now requires userId, like
  getConversation() and getPendingInterrupts() (inferenceOnly agents
  don't use it).
- Document the new errors (README table, stream()/resume() JSDoc),
  StreamOptions.conversationId/userId/channelId and
  AgentStreamResult.channelId, the inferenceOnly caveat, and the
  channel-key change on upgrade; wire resume/getPendingInterrupts in the
  README End-to-End frontend.
- The umbrella @aws-blocks/blocks changeset entry for the Agent change.
- e2e: a conversation-less turn by user B onto user A's id leaves A's
  history unchanged.
From the second review round:
- invokeTurn() scopes the channel inside its try, never reports an
  error on the bare caller-chosen channelId (it could be another user's
  scoped channel), and logs a failed error publish instead of leaving an
  unhandled rejection in the fire-and-forget turn. stream() and resume()
  scope the channel before dispatching, so an unscopable userId fails
  the call itself; stream() no longer leaves its channel promise
  unhandled.
- The AgentCore invocation handler moves to an internal module
  (agentcore-invocation.ts, not in the export map) and is tested with an
  AgentCore session id, so a fallback to it as the conversationId would
  fail a test.
- Docs: InvalidUsage only applies with persistence; mark the
  conversation-less AWS session change as breaking for deployed agents
  that relied on it; DistributedTable read-consistency wording.
From the third review round:
- resume() derives the user-scoped channel before it persists any
  approval, so a rejected resume leaves no approval row behind.
- A userId that cannot be encoded into a channel key throws a branded
  InvalidUsage instead of a raw URIError (which reached clients as a
  nameless 500); the AgentCore session hash for a turn without a
  conversation no longer encodes, so it never throws.
- invokeTurn's JSDoc names the one case that publishes no chunk.
- Changeset title and DESIGN.md wording.
…d writes

From the fourth review round:
- resume() scopes the channel and validates the tool context before the
  ownership read and before persisting approvals, so a rejected resume
  (including an invalid toolContextSchema context) leaves no approval
  row that would hide the still-pending interrupt; stream() scopes the
  channel before its ownership read.
- Tests: the context-rejection case and the conversation-less session
  hash (never throws, keeps (userId, channelId) pairs apart).
- JSDoc: stream()/resume() list InvalidUsage (and resume the other
  errors it throws); invokeTurn names both cases that publish no chunk;
  agent.aws.ts session-id comments describe the conversation-less case.
…g approvals

Comment-only follow-up from the fifth review round: approvals are
persisted after every validation and ownership check, but a dispatch
failure can still reject resume() after they are written. Also list
InvalidModelConfig in stream()'s @throws and reword the AgentCore
session-id comment.
@changeset-bot

changeset-bot Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f4c9d30

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@aws-blocks/bb-agent Minor
@aws-blocks/blocks Minor
@aws-blocks/bb-distributed-table Minor
@aws-blocks/bb-realtime Patch
@aws-blocks/bb-async-job Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Comment thread packages/bb-agent/src/agent.aws.ts Fixed
…sion

A turn without a conversation has no state to keep, so its AgentCore
runtimeSessionId is now a fresh turn-<uuid> instead of a hash of the
user and channel. No two such turns (of the same or different users)
share a session, and the user id no longer flows into a hash (flagged
by code scanning as a password hash).
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor
📖 Glossary & notes — scoring, indicators, per-metric thresholds (click to expand)
  • N = 1 — one rep per cell, so a small delta may be model variance, not a real change; re-run for certainty.
  • Indicators (change vs baseline, per metric): ✨ noticeably improved · ✅ OK / no significant change (within noise band) · ⚠️ slight regression (beyond 1× threshold) · ❌ serious regression (beyond 2× threshold) · 🆕 new — no baseline value yet (current value shown, tagged (new)) · — nothing to show this run · 🗑️ cell gone since the baseline. Each cell shows <indicator> <value> (<Δ vs main>) inline.
  • Columns: Tests (pass/denom) · Judge (overall 0-10 score with signed delta; per-dimension breakdown lives in the judge artifact JSON) · Cost · Turns (agent cycles) · Score · Stop reason.
  • Threshold definitions (per metric, DELTA_THRESHOLDS in overview.mjs):
    • composite/score: ±5 points — a change ≤5 pts is ✅; >5 improving is ✨; >5 worsening is ⚠️; >10 worsening is ❌.
    • judge (overall): ±0.3 — a shift ≤0.3 is ✅; >0.3 improving is ✨; >0.3 worsening is ⚠️; >0.6 worsening is ❌.
    • tests: ±1 pass — a ≤1 pass change is ✅; >1 improving is ✨; >1 worsening is ⚠️; >2 worsening is ❌.
    • cost: ±10% of baseline (floor $0.02) — within 10% is ✅; >10% cheaper is ✨; >10% more expensive is ⚠️; >20% more expensive is ❌.
    • turns: ±3 cycles — a ≤3 turn change is ✅; >3 fewer is ✨; >3 more is ⚠️; >6 more is ❌.
  • Directions: higher is better for tests, judge, and score; lower is better for cost and turns.
  • Composite (0-100) = round(60·test_rate + 4·judge·min(1, 4·test_rate), 1) — 60% objective pass-rate + 40% judge, the judge term gated below a 25% pass-rate.
  • Cost = builder token spend at Bedrock Claude Opus 4.8 rates ($5 in / $25 out per 1M tokens; BUILDER_PRICING in scoring.mjs).
  • SCORE = composite ÷ cost — composite points per dollar (higher = better). Cost, not raw token volume, is the denominator, so tokens can't dominate; a broken (composite 0) cell scores 0 no matter how cheap. Flip SCORE_PER_DOLLAR in scoring.mjs for cost-per-point (lower = better).
  • Baseline = the most recent main-branch bench (bench/runs/latest-main.json), NOT the PR base commit — the PR always diffs against the current state of main.
  • Excluded from the mean: harness_error cells (infra failures) and gradeable cells that ran no tests. agent_fail (agent produced no app in budget) and dead_server (built app never served / crashed) are BOTH included, as composite 0.
  • Run artifacts — per-cell source + full agent traces · 🕒 Last run: 2026-10-23T17:53:11Z · run #39988336442

Results — PR vs main baseline

Each metric shows its current value colored by the change vs the latest main baseline b16e5b6. ✨ improved · ✅ within noise · ⚠️ slight regression · ❌ serious regression · 🆕 no baseline yet (value shown, tagged (new)).

  • ⚠️ All 10 cell(s) were harness_error — nothing was scored, no composite headline.
Task Template Tests Judge Cost Turns Score Stop reason
async-word-counter bare — — ✨ $3.38 (-$3.62) ✨ 21 (-15) — endTurn
auth-notes demo — — ✅ $6.29 (-$0.46) ✅ 32 (0) — endTurn
cognito-profile auth-cognito — — ✨ $9.81 (-$4.42) ✨ 40 (-9) — endTurn
collab-presence-board default — — ✨ $2.16 (-$1.05) ✨ 16 (-5) — endTurn
email-digest demo — — ✅ $5.3 (-$0.1) ⚠️ 31 (+5) — endTurn
file-gallery bare — — ✨ $1.83 (-$1.2) ✨ 14 (-4) — endTurn
kb-chat-agent demo — — ✨ $18.25 (-$5.38) ✨ 46 (-7) — endTurn
observability-api backend — — ✅ $5.17 (+$0.07) ✅ 27 (-2) — endTurn
oidc-dsql-notes react — — ✅ $8.23 (-$0.7) ✅ 35 (0) — endTurn
sql-kb-catalog nextjs — — ✅ $5.59 (-$0.06) ⚠️ 37 (+5) — endTurn

Merge verdict

✅ This bench run does not block the PR.

  • ⚪ Absolute floor — BENCH_MIN_SCORE=65 set, but no cell was scored — floor skipped (conservative).
  • ⚪ Regression vs main — BENCH_MAX_REGRESSION=10 set, but no main baseline to diff — regression gate skipped.

The Δ vs base ball above is a fixed ±5-pt display band; the regression gate uses BENCH_MAX_REGRESSION. The ball is cosmetic — the gate threshold is what blocks.

Caveats & exclusions

  • 🧰 Excluded (harness_error, not scored) — 10: 10× isolation_unavailable. Cells: async-word-counter/bare, auth-notes/demo, cognito-profile/auth-cognito, collab-presence-board/default, email-digest/demo, file-gallery/bare, kb-chat-agent/demo, observability-api/backend, oidc-dsql-notes/react, sql-kb-catalog/nextjs

@belaboe
belaboe deployed to agent-bench October 23, 2026 16:48 — with GitHub Actions Active
@belaboe
belaboe deployed to agent-bench October 23, 2026 16:48 — with GitHub Actions Active
@belaboe
belaboe deployed to agent-bench October 23, 2026 16:48 — with GitHub Actions Active
@belaboe
belaboe deployed to agent-bench October 23, 2026 16:48 — with GitHub Actions Active
@belaboe
belaboe deployed to agent-bench October 23, 2026 16:48 — with GitHub Actions Active
@belaboe
belaboe deployed to agent-bench October 23, 2026 17:41 — with GitHub Actions Active

This branch had an error being deployed

1 failed and 1 active deployments
agent-bench — f4c9d305 Deployed Oct 23, 2026 by belaboe via bench / summary #3352
e2e — f4c9d305 Deployed Oct 23, 2026 by belaboe via E2E Compute Smoke #3999
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants