Skip to content

Fix three demo bugs: browser beside apps, doubled tool calls, vendor names in steps - #824

Merged
guidovizoso merged 14 commits into
mainfrom
guido/browser-tools-beside-plugins
Oct 22, 2026
Merged

guidovizoso merged 14 commits into
mainfrom
guido/browser-tools-beside-plugins

Conversation

@guidovizoso

@guidovizoso guidovizoso commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Three fixes found while running the webinar demo flows on a fresh stack from main.

1. A Bot with an app or web research could not use its browser in a conversation

A Bot that held any app or research tool (Parallel Search, a Composio app, and so on) could not use its browser in a conversation. Every computer_* step came back as The tool computer_navigate is unavailable for this headless turn., and the Bot fell back to fetching the page some other way. A Bot holding no such tools was unaffected.

Cause. In a conversation the browser offers the computer_* tools itself. When the Bot also holds server-side tools, server/src/copilot.ts wraps the run in HeadlessToolsMiddleware. The middleware claimed every pending call whose name starts with computer_ (a rule from #673 so an unattended turn answers an unknown computer tool with a refusal). It holds no computer tools in this case, so it refused each one before the browser could run it.

Fix. toolkit/src/headless-tools.ts now claims a computer_* call only when the caller did not offer that tool. A call the browser offered goes back to the browser, through the mixed-ownership path the middleware already had. Unattended turns offer no client tools, so they behave as before.

Tests. A new case in server/tests/headless-tools.test.ts gives the middleware a plugin tool while the caller offers computer_navigate. The middleware must answer nothing, make no second request and finish the run. It failed before the change.

2. Every tool call was stored, and drawn, twice

When a Bot reached for an app the person had not connected (Airtable via Composio), the conversation showed two identical Connect cards, and still did after a reload.

Cause. agent-langgraph/src/stream.ts sent TOOL_CALL_START with no parentMessageId. The browser therefore named each call's message after the call (call_1), while the conversation store kept the step as assistant-call_1. Two things followed:

  • mergeStoredMessages in app/src/components/channels/channel-chat.tsx matched stored messages to live ones by id only, so it inserted the stored copy as missing.
  • The browser sent its own call_1 copy back with the next message, and the store kept both.

Fix.

  • agent-langgraph gives each step's calls the store's name for their message, taken from the step's first call, so a new conversation stores each call once.
  • mergeStoredMessages treats a stored assistant message whose every call one live message already holds as that message, so conversations that already have the copy draw each call once.

Tests. agent-langgraph/tests/stream.test.ts pins parentMessageId for a step that ran two tools and for a call the surface owns. app/tests/merge-stored-messages.test.ts covers three merge cases: the renamed copy is not added, a missing message still lands before the live message holding the call, and an unknown call is still added. Every one of these tests that exercises the changed behaviour failed before its change.

3. An app's steps are named for the app, with no vendor prefix

A Composio app's steps read "Used Composio Airtable", then "Airtable list bases", and an Arcade app's key reached the screen as arcade-Gmail, because the labels only knew Composio's prefix.

Fix. app/src/lib/plugins/tool-name.ts now drops either broker's prefix (composio-, arcade-). It starts a brokered action at the verb and names the app once, beside it. tool-activity.tsx uses the same connectorName, so the summary reads "Used Airtable" and a step reads "List bases · Airtable". Labels for servers that aren't brokered are unchanged.

Tests. app/tests/tool-name.test.ts and app/tests/tool-activity.test.tsx cover Composio and Arcade actions and the summary line.

Checked

By hand on a fresh local stack:

  • A Bot granted Parallel Search browses Hacker News. Before the fix, no computer.* audit row was written. After it, computer_navigate is carried out and the Computer pane opens.
  • An Airtable question on an unconnected account draws one Connect card, live and after a reload.
  • After the agent rebuild, a new two-turn Airtable conversation stores each call exactly once.
  • It reads "Used Airtable → List bases · Airtable, Get base schema · Airtable", with no "Composio" anywhere on the page.

The following are all clean:

  • Format, lint, and typecheck (app, server, toolkit)
  • Tests: bun test ./app (1595), toolkit/tests (59), agent-langgraph (42), server/tests (5734 pass, 19 skip)
  • bun run build and the agent-langgraph Docker build

…s plugin tools

In a conversation the browser offers the computer_* tools itself. When the Bot also held an app or research tool, the run was wrapped in HeadlessToolsMiddleware, which claimed every computer_* call and, holding no such tool, answered each one as unavailable for this headless turn. It now claims a computer_* call only when nobody else offered that tool, so an unattended turn still answers an unknown one with a refusal.
The stored conversation names a tool call's message differently from the live stream (assistant-call_1 where the stream said call_1), and mergeStoredMessages matched on message id only, so it inserted the stored copy as missing. Every tool call that came back from the store that way was drawn twice, which is most visible as two Connect cards for one refusal. A stored assistant message whose every call one live message already holds now anchors on that message instead of being added.
@guidovizoso guidovizoso changed the title Let a Bot with an app or web research use its browser in a conversation Let a Bot with apps use its browser in a conversation, and draw one Connect card per call Oct 22, 2026
Composio publishes every action as <APP>_<VERB>_<NOUN>, so a step read "Airtable list bases", and an Arcade app's key reached the screen as arcade-Gmail because only the composio- prefix was known. A brokered app's action now starts at the verb, "List bases", with the app named once beside it, and connectorName drops either broker's prefix.
agent-langgraph sent TOOL_CALL_START with no parentMessageId, so the surface named each call's message after the call (call_1) while the conversation store kept the step as assistant-call_1. The surface then sent its copy back on the next turn and the store kept both, so every tool call of a finished turn was stored, and drawn, twice. Each step's calls now carry the store's name for their message, taken from the step's first call.
@guidovizoso guidovizoso changed the title Let a Bot with apps use its browser in a conversation, and draw one Connect card per call Fix four demo bugs: browser beside apps, doubled tool calls, vendor names in steps Oct 22, 2026
@guidovizoso guidovizoso changed the title Fix four demo bugs: browser beside apps, doubled tool calls, vendor names in steps Fix three demo bugs: browser beside apps, doubled tool calls, vendor names in steps Oct 22, 2026
@guidovizoso
guidovizoso merged commit e7916b3 into main Oct 22, 2026
19 checks passed
@guidovizoso
guidovizoso deleted the guido/browser-tools-beside-plugins branch October 22, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant