Add RCA failure clustering tools and build insights improvements - #382
Merged
Conversation
Pure, dependency-free parser for TFA reply_markdown: reads the additive
<!-- tfa-status: {...} --> marker when present, infers RESOLVED from a
## Final RCA heading otherwise, normalizes confidence (string + numeric),
extracts tasks/questions, and never throws on malformed input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stateless submit+poll against the o11y rcaChat proxy. Hardcoded o11y base const (no env, no localhost), apiClient (not fetch), manual Basic auth header, raise_error:false on the poll, progress notifications, and clean group-scope- safe error mapping (consent/IDOR/expired/failed) plus a soft-PENDING handoff on wall-clock cap. Zod param shapes live in constants.ts; no credential fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single team-agnostic submit+poll-in-one MCP tool wired behind a minimal Zod schema and the real inline trackMCP (4-arg) + try/catch + handleMCPError pattern (no nonexistent withInstrumentation middleware). Domain failures return a specific isError envelope and still fire exactly one trackMCP. Registered in the server-factory toolAdders array. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tfaRcaTurn tool now reads the LLM-enforced TurnResponse (status/confidence/asks/questions/suggestions/hypotheses/rca/blocked) directly from the o11y rcaChat poll body instead of inferring status from free markdown + the <!-- tfa-status --> HTML-comment marker. - delete src/tools/tfa-rca-utils/status-parser.ts + its test - types.ts mirrors the misc-services TurnResponse (TfaAsk/EvidenceType/TfaRca) - submit-turn.ts maps the nested structured turn (snake_case -> camelCase), drops replyMarkdown/tasks/marker handling - tfa-rca-collaboration.ts returns the structured fields; PENDING still carries turnId for resume - tests assert structured reads (typed asks routable by evidenceType, RESOLVED rca, BLOCKED reason/unmetAsks, status-without-inference); inference/marker assertions removed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the rcaChat o11y base URL from the process-startup config singleton (src/config.ts O11Y_TFA_RCA_BASE_URL, env-derived and frozen) instead of a hardcoded constant, so the tool can target the o11y env where a build's representatives live. Default stays the rengg-tfa endpoint, so existing callers are unchanged. Not a process.env read in tool code and not a Zod tool input field, per tool-design + multi-tenant-safety + security rules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface per-test failure detail (category, capped error summary, file path, flaky flags) behind a new includeFailureDetail flag (default false → byte-identical existing payload). Reads sibling fields already returned by the test-runs API and discarded today; never returns full stack traces. Seeds downstream failure-signature clustering without per-test probe turns. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ature Closes the coverage gap flagged in review: error_summary extraction from an object entry using the .text fallback (alongside the existing .message case). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ging)
getTestIds hardcoded api-automation.browserstack.com, so it could not list a
build that lives in a non-prod env (e.g. rengg-tfa). Add a
BROWSERSTACK_AUTOMATION_BASE_URL startup-config knob (default prod), mirroring
the O11Y_TFA_RCA_BASE_URL pattern, surfaced via getAutomationBaseUrl(); listTestIds
now builds its /ext/v1/builds/{id}/testRuns URL from it. Set the env var to a
rengg/staging host to point the tool there — no code change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… list extractFailedTestIds gated on 'status===status && run_count', but JUnit-uploaded builds report run_count=0 even for genuinely failed tests, so listTestIds returned zero rows for them (the build looked empty though the API clearly had failures). The observability_url 'details=<id>' check already filters to real test nodes (suite/hook nodes carry no status/url), so the run_count guard was redundant and harmful. Drop it. Verified against a real rengg-tfa upload: 0 -> 7 failed rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l output
triggerRcaReport POSTs the o11y external Release Readiness trigger
(/ext/v1/ai/builds/{buildUuid}/releaseReadiness/trigger?force=<bool>)
with Basic auth and returns a trimmed glimpse only ({state, verdict,
verdictProvisional, partial, analyzedCount, totalFailedCount, totalPrs,
faultyPrNumbers, failureReason, viewReport}) — never the raw response
or the prs[]/workflows[] arrays. 403 plan/flag fences and
REPO_NOT_CONFIGURED / RELEASE_READINESS_NOT_FOUND map to clear isError
texts. viewReport links the TRA UI via new startup config
BROWSERSTACK_O11Y_UI_BASE_URL (default observability.browserstack.com);
the /builds/{buildUuid} path shape is flagged as an assumption behind a
single constant.
tfaRcaTurn terminal turns are now trimmed in submit-turn: RESOLVED
returns {status, confidence, threadId, glimpse{root_cause <=220 chars,
failure_type, related_prs}, viewRca} and drops the full rca payload;
NEEDS_INFO keeps questions/asks/suggestions/hypotheses verbatim for the
client loop; PENDING keeps {status, turnId, threadId}. The full RCA
lives on the Test Observability dashboard — the tool surfaces a glimpse
and redirects humans there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
viewReport/viewRca now point at the real AI-TFA report location: https://automation.browserstack.com/builds/{buildUuid}?tab=ai_report&subTab=aitfa (designer-confirmed 2026-07-13; replaces the flagged observability.browserstack.com /builds/{buildUuid} assumption). {buildUuid} is substituted from the caller-given build id. Default BROWSERSTACK_O11Y_UI_BASE_URL updated; tests pinned to the new shape. npm run build green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idge # Conflicts: # src/tools/rca-agent-utils/get-failed-test-id.ts
api-automation.browserstack.com — live-verified to route the /ext/v1 testRca + rcaChat endpoints (probe returned the persisted E2E RCA). Staging tenants (api-observability-<tenant>.bsstag.com) become the override case. Build gate green (265 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idge Feat/tfa rca mcp bridge
Previously listTestIds only worked when a status filter was passed — calling it
without one returned nothing (extraction matched node.details.status === status,
which is false for every real test when status is undefined). Now:
- No status → every real test node is returned, each carrying its own status
(passed/failed/pending/skipped). Same obs-api endpoint (/ext/v1/builds/{id}/
testRuns), which already returns all statuses when test_statuses is omitted.
- Optional status still narrows the result (server query + extraction), so
callers like the rca-build skill that pass status='failed' are unchanged.
- FailedTestInfo gains ; failure signatures attach only to FAILED tests.
- extractFailedTestIds kept as an alias of the renamed extractTestIds.
Build gate green (267 tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idge feat(rca): listTestIds returns ALL tests of a build (each with status)
…t 5 pages) The old requestNumber>=5 cap dropped tests for any build with >5 pages — e.g. build 11fdi8f5… returned 117 tests/7 failed when the true totals are 171/9 (8 pages). Bumped to MAX_PAGES=100 and log a warning if ever hit, so results are never silently partial. Now returns all 171 tests / 9 failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idge fix(rca): raise listTestIds pagination cap (was silently truncating a…
`tfaRcaTurn` submits and polls to completion; when the wall-clock cap is
hit it returns a soft PENDING carrying turnId. Reading that turn back
previously meant re-entering the polling tool, which blocks the caller
for up to another 90s.
Add `getTfaTurnResult({testRunId, turnId})`: a single GET on the rcaChat
poll path — no submit, no loop — returning the same trimmed,
status-discriminated contract (NEEDS_INFO asks/questions verbatim,
RESOLVED glimpse + viewRca, BLOCKED reason/unmetAsks), or PENDING when
the agent is still working so the caller decides when to read again.
The read/mapping layer (readStructuredTurn, toTrimmedResult, auth header,
poll URL, TfaRcaTurnError) moves verbatim into tfa-rca-utils/turn-result.ts
and is now shared by both tools. `submitTfaRcaTurn` keeps its polling loop
unchanged and re-exports TfaRcaTurnError, so importers are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…idge feat(rca): add getTfaTurnResult one-shot turn read
Wires the build-level failure-theme clustering API (trigger + poll, paginated test membership) into the MCP server for the rca-build pipeline's server-side clustering step. Falls back to ready:false if clustering exceeds the poll budget or the trigger endpoint is unavailable, so a caller can fall back to client-side clustering instead of blocking. Also fixes CustomFieldValue formatting in create-testcase.ts.
…emantics Per Anthropic's tool-definition best practices (3-4 sentences per description, explicit "do NOT call this if X" guidance to prevent redundant calls). Adds thread-continuation rules to tfaRcaTurn/getTfaTurnResult (threadId semantics, PENDING polling via getTfaTurnResult instead of resubmitting), idempotency/force semantics to triggerRcaReport, and call-once-per-build + pagination guidance to getBuildFailureThemes/ listTestsInFailureTheme.
…-result.ts Orphaned by the prior raw-rca-passthrough temp fix, which stopped calling the trimming helper. Removing the now-dead symbols only; the passthrough behavior itself is left as-is.
toTrimmedResult's RESOLVED branch was passing the full raw rca object through (a "temp fix ... to remove later") instead of populating glimpse, silently breaking two things: the 3 tests asserting the trimmed contract, and rca-build's Step 4b fast-path, which reads glimpse verbatim off a RESOLVED turn to flip the CSV row and seed siblings without a Step 5 dispatch. Restores truncate(root_cause, 220)/failure_type/related_prs via glimpse, drops the raw passthrough.
…e signal
The POST /ext/v1/ai/failures/{buildUuid} trigger endpoint is deployed, so a
never-computed build now reaches ready:true on its own. Update the doc comments
to describe ready:false / trigger-unavailable as a genuine server outage rather
than the routine path for a fresh build. Comment-only; no logic change.
… metadata Expose o11y vcs_info (branch/sha/name) in the fetchBuildInsights tool output so callers can resolve the build's actual working branch instead of assuming the current git branch.
…idation (#373) * fix(rca): mandate PR details as context to TFA agent with contract validation AIR-607: incomplete/missing PR context was misleading the TFA agent. Introduce a typed PrDetail contract (title, author, link, number, tag=latent|regression) on the tfaRcaTurn tool; validate it when present (reject partial PR objects with a clear error rather than forwarding them); and always concatenate a stringified PR_DETAILS block onto the message sent to TFA (explicit 'none provided' when absent) so PR context is never silently dropped. * fix(rca): make PR identity repo+number in prDetails contract Add required repo (owner/name) field and enforce link is the canonical https://github.com/<repo>/pull/<number>. A bare PR number is unique only within its repo, so number-only identity let PRs from different repos collide into one card (AIR-607 report showed the same #861 across unrelated cases + a 404 link). * test(rca): assert prDetails in tfaRcaTurn field list; lint prDetails describe
Remove over-explained JSDoc/block comments that narrated reasoning or restated obvious behavior; keep only short comments where genuinely useful.
Not wired into server-factory for now; will be enabled only for remote MCP server users.
ruturaj-browserstack
changed the title
Lets consumers do `import { addTfaRcaCollaborationTools } from "@browserstack/mcp-server"` instead of reaching into dist/tools/tfa-rca-collaboration.js.
SavioBS629
previously approved these changes
Aug 31, 2026
gaurav-singh-9227
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds RCA (root cause analysis) failure-clustering tools on top of the existing TFA RCA integration, plus small build-insights improvements:
getBuildFailureThemes/listTestsInFailureTheme— clusters a build's failed tests into failure themes/root-cause groups, and lists the tests belonging to a given theme.listTestIds— now returns all tests in a build (with status), removing the previous pagination cap that silently truncated results.triggerRcaReport— triggers/links the RCA report view for a build.branch,commit_sha, andvcs_namefrom build metadata.Test plan
npm run typechecknpm testgetBuildFailureThemes,listTestsInFailureTheme,listTestIds, andtriggerRcaReportagainst a real build