Skip to content

test(core): review fixes for #896 - guard destroy and cdk ls --app quoting - #1014

Open
belaboe wants to merge 4 commits into
belaboe/fix-windows-sandbox-destroyfrom
review/896-fixes
Open

belaboe wants to merge 4 commits into
belaboe/fix-windows-sandbox-destroyfrom
review/896-fixes

Conversation

@belaboe

@belaboe belaboe commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Review fixes for #896 (fix(core): make sandbox:destroy work on Windows). Stacked on belaboe/fix-windows-sandbox-destroy; merging this PR folds the fixes into #896. It merges cleanly with current main (no rebase needed).

What it adds:

  • Destroy and cdk ls argv move into buildSandboxDestroyArgs / buildSandboxLsArgs. Their --app value is asserted with literals for cmd.exe (double quotes) and POSIX (single quotes, '\'' escaping), so a revert to '${backendPath}' fails a unit test on a Linux runner. (review comment r4359557460)
  • buildSandboxSynthArgs / buildSandboxDeployArgs take an optional platform, and their Windows and POSIX --app forms are asserted the same way.
  • sandbox-stack-listing.test.ts replaces run-command in a child process and asserts that both cdk ls listers use runCapture (npm / npx) and both destroys use runSync with --conditions=cdk. The fixture path contains a single quote, so re-inlining the old argv at a call site also fails. listSandboxStackNames and listProductionStackNames are exported as @internal for this; scripts/index.ts and the package export map are unchanged.
  • quoteCdkAppPath refuses a " on Windows (suggestion applied, with a test, a @throws note and a changeset line). (r4359557500)
  • A POSIX-only SIGTERM test for runCapture's signal branch (suggestion applied). (r4359557531)

Each new test was checked against a mutation of the line it guards (10 mutations, all red; all green on this head).

Not in this PR:

Windows proof: Windows E2E run 39972013006 on this head (result in a comment once it finishes). The scheduled job's status is not trustworthy for destroy until #946 lands, so the destroy logs were read directly.

PR Checks do not run for this PR because its base branch name contains / (pull_request: branches: ['*']). Local lanes (build, lint:deps, unit tests, e2e local, lint, check:api) were run on this head.

…n wiring

Pull the sandbox destroy and cdk ls argv into buildSandboxDestroyArgs and
buildSandboxLsArgs and assert their --app value with literals for both
cmd.exe and POSIX, so a revert to single quotes (the original Windows bug)
fails a test. A module-mock test asserts that both destroy paths list their
stacks with runCapture and run cdk destroy with runSync. quoteCdkAppPath now
refuses a double quote on Windows, and runCapture's signal branch has a test.
… wiring test

buildSandboxSynthArgs and buildSandboxDeployArgs take an optional platform,
so their win32 and POSIX --app forms are asserted with literals. The wiring
test uses a path with a single quote (so re-inlining the old single-quoted
argv at a call site fails), asserts the argv literally, and checks that both
destroys run CDK with --conditions=cdk. listSandboxStackNames builds its argv
outside the try, so an invalid path is not reported as a listing failure.
@changeset-bot

changeset-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b686227

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 2 packages
Name Type
@aws-blocks/core Patch
@aws-blocks/blocks Patch

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@belaboe

belaboe commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Windows E2E on head b686227: run 39972013006. From the logs (the job status alone is not proof until #946): the first sandbox:destroy attempt hit an API Gateway 429 on DeleteRestApi, the retry listed the stacks through cdk ls with no ENOENT and no listing warning, emptied the buckets and ended with "Sandbox destroyed on retry!"; npm run destroy ended with "Production stack destroyed!". No Cannot find module error.

@github-actions

Copy link
Copy Markdown
Contributor
📖 Glossary & notes — scoring, indicators, per-metric thresholds (click to expand)
  • N = 1 — one rep per cell, so a small delta may be model variance, not a real change; re-run for certainty.
  • Indicators (change vs baseline, per metric): ✨ noticeably improved · ✅ OK / no significant change (within noise band) · ⚠️ slight regression (beyond 1× threshold) · ❌ serious regression (beyond 2× threshold) · 🆕 new — no baseline value yet (current value shown, tagged (new)) · — nothing to show this run · 🗑️ cell gone since the baseline. Each cell shows <indicator> <value> (<Δ vs main>) inline.
  • Columns: Tests (pass/denom) · Judge (overall 0-10 score with signed delta; per-dimension breakdown lives in the judge artifact JSON) · Cost · Turns (agent cycles) · Score · Stop reason.
  • Threshold definitions (per metric, DELTA_THRESHOLDS in overview.mjs):
    • composite/score: ±5 points — a change ≤5 pts is ✅; >5 improving is ✨; >5 worsening is ⚠️; >10 worsening is ❌.
    • judge (overall): ±0.3 — a shift ≤0.3 is ✅; >0.3 improving is ✨; >0.3 worsening is ⚠️; >0.6 worsening is ❌.
    • tests: ±1 pass — a ≤1 pass change is ✅; >1 improving is ✨; >1 worsening is ⚠️; >2 worsening is ❌.
    • cost: ±10% of baseline (floor $0.02) — within 10% is ✅; >10% cheaper is ✨; >10% more expensive is ⚠️; >20% more expensive is ❌.
    • turns: ±3 cycles — a ≤3 turn change is ✅; >3 fewer is ✨; >3 more is ⚠️; >6 more is ❌.
  • Directions: higher is better for tests, judge, and score; lower is better for cost and turns.
  • Composite (0-100) = round(60·test_rate + 4·judge·min(1, 4·test_rate), 1) — 60% objective pass-rate + 40% judge, the judge term gated below a 25% pass-rate.
  • Cost = builder token spend at Bedrock Claude Opus 4.8 rates ($5 in / $25 out per 1M tokens; BUILDER_PRICING in scoring.mjs).
  • SCORE = composite ÷ cost — composite points per dollar (higher = better). Cost, not raw token volume, is the denominator, so tokens can't dominate; a broken (composite 0) cell scores 0 no matter how cheap. Flip SCORE_PER_DOLLAR in scoring.mjs for cost-per-point (lower = better).
  • Baseline = the most recent main-branch bench (bench/runs/latest-main.json), NOT the PR base commit — the PR always diffs against the current state of main.
  • Excluded from the mean: harness_error cells (infra failures) and gradeable cells that ran no tests. agent_fail (agent produced no app in budget) and dead_server (built app never served / crashed) are BOTH included, as composite 0.
  • Run artifacts — per-cell source + full agent traces · 🕒 Last run: 2026-10-23T15:43:12Z · run #39975742907

Results — PR vs main baseline

Each metric shows its current value colored by the change vs the latest main baseline b16e5b6. ✨ improved · ✅ within noise · ⚠️ slight regression · ❌ serious regression · 🆕 no baseline yet (value shown, tagged (new)).

  • ⚠️ All 10 cell(s) were harness_error — nothing was scored, no composite headline.
Task Template Tests Judge Cost Turns Score Stop reason
async-word-counter bare — — ✨ $5.73 (-$1.28) ✨ 28 (-8) — endTurn
auth-notes demo — — ❌ $10.25 (+$3.5) ❌ 47 (+15) — endTurn
cognito-profile auth-cognito — — ✨ $11.17 (-$3.06) ✨ 39 (-10) — endTurn
collab-presence-board default — — ❌ $7.36 (+$4.15) ❌ 37 (+16) — endTurn
email-digest demo — — ✨ $4.64 (-$0.76) ✅ 24 (-2) — endTurn
file-gallery bare — — ✨ $2.71 (-$0.32) ✅ 19 (+1) — endTurn
kb-chat-agent demo — — ❌ $32.06 (+$8.43) ❌ 65 (+12) — endTurn
observability-api backend — — ❌ $6.38 (+$1.28) ✅ 30 (+1) — endTurn
oidc-dsql-notes react — — ✨ $6.59 (-$2.34) ✨ 29 (-6) — endTurn
sql-kb-catalog nextjs — — ✨ $4.64 (-$1.01) ✨ 27 (-5) — endTurn

Merge verdict

✅ This bench run does not block the PR.

  • ⚪ Absolute floor — BENCH_MIN_SCORE=65 set, but no cell was scored — floor skipped (conservative).
  • ⚪ Regression vs main — BENCH_MAX_REGRESSION=10 set, but no main baseline to diff — regression gate skipped.

The Δ vs base ball above is a fixed ±5-pt display band; the regression gate uses BENCH_MAX_REGRESSION. The ball is cosmetic — the gate threshold is what blocks.

Caveats & exclusions

  • 🧰 Excluded (harness_error, not scored) — 10: 10× isolation_unavailable. Cells: async-word-counter/bare, auth-notes/demo, cognito-profile/auth-cognito, collab-presence-board/default, email-digest/demo, file-gallery/bare, kb-chat-agent/demo, observability-api/backend, oidc-dsql-notes/react, sql-kb-catalog/nextjs

@belaboe
belaboe marked this pull request as ready for review October 23, 2026 16:53
@belaboe
belaboe requested a review from a team as a code owner October 23, 2026 16:53

This branch was previously deployed

2 inactive deployments
agent-bench — b686227b Deployed Oct 23, 2026 by belaboe via bench / summary #3342
e2e — b686227b Deployed Oct 23, 2026 by belaboe via Windows E2E (dev + sandbox + deploy) #274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant