Skip to content

feat(agents): conformance scenarios for agent discovery (agents-wg#22) - #556

Open
panyam wants to merge 4 commits into
modelcontextprotocol:mainfrom
panyam:feat/agents-conformance-suite
Open

panyam wants to merge 4 commits into
modelcontextprotocol:mainfrom
panyam:feat/agents-conformance-suite

Conversation

@panyam

@panyam panyam commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #576. agents-wg#22 merged on 2026-10-20, so this is ready for review.

What changes

Adds a requirement-traceability yaml for the agent discovery extension, agents-wg#22, and two server scenarios that grade an implementation against it, plus 40 negative controls.

Scenario What it grades Checks
sep-9998-agents-discovery the io.modelcontextprotocol/agents capability; agents/list shape, cards and pagination; the flat agents/get result and modelRoutingHint; every returned tool matching its tools/list entry; agent_not_found and invalid_agent_definition; a hidden agent indistinguishable from an unknown one 27
sep-9998-agents-list-changed notifications/agents/list_changed over subscriptions/listen with agentsListChanged, and its acknowledgement 4

The yaml declares 49 rows quoting the proposal at its merge commit 137d88bd, each with a url to its section, checked by script against the pinned file. 31 are graded. The other 18 are declared and excluded, each with its reason: client and host obligations, rules that compare two authorization contexts (a run holds one credential), and step-up, a MAY that needs a credential the harness can raise.

Things to look at first

  • The number is a placeholder. The proposal lives in the WG repository, so it has no modelcontextprotocol/modelcontextprotocol PR number, and the traceability tooling drops non-numeric ids. The yaml header explains sep-9998 (9999 was the MCP Events placeholder before SEP-3415 existed) and the scripted rename once a number exists.
  • Two optional diagnostic controls. The hidden-agent, broken-agent and list_changed rows need the server to provoke a state over the wire. The scenarios look for agents_conformance_fixture and agents_conformance_add_agent in tools/list, and report untestable without them, the same pattern as feat(sep-3415): Conformance suite for triggers and events #521.
  • --force. The extension isn't on the spec timeline, so the runner selects these scenarios only with --force.
  • ttlMs and cacheScope are graded as required, because both results are CacheableResult, where the draft schema requires them.
  • No -32021 row yet. Luca said on Discord that a client that didn't declare the extension should get -32021. The proposal text doesn't say so yet, so I'll add the row once that follow-up merges.

Evidence

Against mcpkit's implementation (examples/agents/discovery --conformance-agents), the result is 28/28 and 4/4. Breaking that implementation four ways, by showing the hidden agent, never sending list_changed, returning a wrong agent_not_found reason, or omitting ttlMs, turns the matching rows red. Running against the Python reference fork is next, once it serves the extension.

npm run build, npm run lint and npm test pass.

Prerequisite knowledge

  • agents-wg#22 at 137d88bd is the text every row quotes. Retrieving Agent Details, Error Handling and Security Implications carry most of the graded rows.
  • AGENTS.md § Scenario design and § Check conventions explain why this is two scenarios, and why a missing control fails rather than skips.
  • feat(sep-3415): Conformance suite for triggers and events #521 shows the same shape for MCP Events, including the diagnostic-control pattern.

…ceholder sep-9998)

agents-wg PR 22 has no number in modelcontextprotocol/modelcontextprotocol
yet, and both traceability filters are numeric, so the rows and check IDs
carry the placeholder 9998. The header explains why and how to rename.

31 graded rows and 18 excluded. Every row quotes the proposal at 9b16840e,
matched with line wrapping ignored, and points at the section it sits
under.
…ged scenarios

Discovery grades the capability, the roster (shape, paging, cache hints,
cards, unique names), the details for every listed agent, tool parity with
tools/list, and the two error reasons. The hidden-agent and broken-agent
rows read their names from the server's agents_conformance_fixture control
and report untestable without it.

List-changed opens a subscriptions/listen stream with agentsListChanged,
grades the acknowledgement either way round, changes the roster through
agents_conformance_add_agent, and waits for the notification and the
re-listed roster.

Registered as pending, since the everything-server does not implement the
extension.
A stateless fixture built per test, conformant by default, with one option
per rule. 40 tests: a conformant run with every check passing, then one
broken rule per test asserting its check flips.
@pkg-pr-new

pkg-pr-new Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@556

commit: 9157913

agents-wg#22 merged on 2026-10-20 as 137d88bd. The proposal file is
byte-identical to head 9b16840e (blob 7534146f), so every row keeps its
quote and only the citation moves.
Comment on lines +354 to +355
if (typeof card.name !== 'string' || card.name === '') {
nameErrs.push(`${label}: name is ${describeValue(card.name)}`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically the spec doesn't say agent names need to be non-empty, though that does make sense. Probably worth correcting the extension spec for this one.

Comment on lines +603 to +607
if (got.error.message !== unknownFailure.message) {
errs.push(
`message ${JSON.stringify(got.error.message)} for the hidden agent, ${JSON.stringify(unknownFailure.message)} for an unknown name`
);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to do an exact match on the error message here? Seems unreasonably strict to disallow e.g. agent "<requested name>" not found (might warrant a spec adjustment)

Comment on lines +215 to +236
/** Every tool `tools/list` serves, following cursors. */
export async function toolsListAll(
conn: Connection,
maxPages = 50
): Promise<Record<string, unknown>[] | RpcFailure> {
const out: Record<string, unknown>[] = [];
let cursor: string | undefined;
for (let i = 0; i < maxPages; i++) {
let res: { tools?: unknown; nextCursor?: unknown };
try {
res = await conn.request('tools/list', cursor ? { cursor } : {});
} catch (err) {
return toRpcFailure(err);
}
for (const t of Array.isArray(res.tools) ? res.tools : []) {
if (isObject(t)) out.push(t);
}
if (typeof res.nextCursor !== 'string' || res.nextCursor === '') break;
cursor = res.nextCursor;
}
return out;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feels like this should report tool list truncation and flag the server as untestable (a la agentsListAll) rather than silently using an incomplete tool list.

Comment on lines +145 to +151
if (!ack) {
const why = session.error
? `${SUBSCRIPTIONS_LISTEN_METHOD} failed: ${session.error.code} ${session.error.message}`
: `${SUBSCRIPTIONS_LISTEN_METHOD} did not confirm with ${SUBSCRIPTIONS_ACKNOWLEDGED_NOTIFICATION} within ${LISTEN_ACK_MS}ms (HTTP ${session.status}, ${session.contentType ?? 'no content type'})`;
if (!declaredTrue) {
return [
untestable(NOT_DECLARED_ID, why),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a server doesn't declare listChanged and answers subscriptions/listen with -32601, or if it doesn't serve it at all, isn't it already not acknowledging the filter key? I think this should be brought in line with the precedent here which marks this case as skipped.

Comment on lines +228 to +231
const hit = await session.waitFor(
(n) => n.method === AGENTS_LIST_CHANGED_NOTIFICATION,
LIST_CHANGED_MS
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this waitFor will pass on servers that happen to emit a list_changed notification immediately upon connection - I believe I've seen servers do that with tools sometimes. Might be worth having an overload that only considers notifications starting from an arbitrary execution breakpoint, or something? Might be overengineering, though.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent discovery extension (agents-wg#22): server conformance scenarios

2 participants