Skip to content

feat(sep-2640): grade 13 more rows, including two-server host rules - #590

Draft
panyam wants to merge 1 commit into
modelcontextprotocol:mainfrom
panyam:feat/sep-2640-coverage
Draft

panyam wants to merge 1 commit into
modelcontextprotocol:mainfrom
panyam:feat/sep-2640-coverage

Conversation

@panyam

@panyam panyam commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

What changes

Grades 13 more sep-2640.yaml rows. The yaml counted 46 of its 89 checks as emitted, and now counts 59.

Six server checks join the existing scenarios:

  • enumeration.ts: sep-2640-resources-nested-completeness and sep-2640-nested-publication-flat for nested skills, plus sep-2640-error-standard-codes (-32602 for a skill file the server does not serve) and sep-2640-error-informative-messages (a WARNING when the message only names the code).
  • directory.ts: sep-2640-directory-read-cache-attributes (ttlMs and cacheScope on the result, from 2026-07-28) and sep-2640-directory-read-empty-dir (an empty directory reads as resources: []).

Four client scenarios, in a new src/scenarios/client/skills/hosting.ts, add seven checks:

Scenario Checks
sep-2640-client-load-by-uri host-load-by-uri, host-no-empty-listing-assumption
sep-2640-client-invalid-entry resources-invalid-entry
sep-2640-client-directory-unlisted-child directory-read-not-manifest-extension
sep-2640-client-two-origins host-skill-identity-pair, host-cross-origin-no-shadow, host-reads-bound-to-origin

The two-origins scenario starts a second mock server and hands its URL to the client in MCP_CONFORMANCE_CONTEXT as secondServerUrl, the way the auth scenarios hand over an authorization server. The runner needed no change.

Prerequisite knowledge

  • AGENTS.md: fewer scenarios with more checks, one ID for SUCCESS and FAILURE, and a negative control per check.
  • The Skills beta page, § Resources, § Nested Skills, § Reading Directories, § Error Handling and § Security Considerations.
  • verification.ts's header, on why the unlisted variant was retired. The new client scenarios are built to avoid the same problem.

Reviewer's guide

The client scenarios grade requests that must not arrive, so two rules shape them.

  • The client library refuses, not the driver. Each contract tells the client to try every load: every listed entry, every file a directory lists. A conformant client refuses the invalid ones. A driver told not to ask, as in the retired unlisted variant, would only grade the driver.
  • Absence needs a prerequisite. Every scenario requires the request a conformant client does make (the valid entry's SKILL.md, the listed supporting file, skills/list on both servers) and reports untestable without it, per Standardize how scenarios report setup/execution failures #248.

Read in this order:

  1. src/scenarios/client/skills/mock-host.ts: one configurable SEP-2640 server that records every request.
  2. src/scenarios/client/skills/hosting.ts: the four scenarios, each with its contract in the class comment.
  3. src/scenarios/client/skills/hosting.test.ts: a conforming scripted client and one rule-breaking client per check. The two-origins tests break it three ways: caching by uri alone, keying skills by name alone, and reading one server's files from the other.
  4. enumeration.ts (nestedChecks, errorChecks) and directory.ts (cacheCheck, emptyDirCheck), then src/scenarios/server/skills/coverage.test.ts.
  5. src/seps/sep-2640.yaml: the header's coverage counts and backing-scenario list. No row changes.

Decision log

  • Nesting and empty directories SKIP when absent. Both are optional for a server. The empty-directory search walks every listed skill's tree, up to 32 directory reads.
  • -32002 also passes error-standard-codes on 2025-11-25. That base spec named -32002 for an unknown resource. From 2026-07-28 only -32602 passes.
  • nested-publication-flat checks that no entry embeds another. Of "nothing in the listing marks nesting", that is the part a harness can see.
  • Left ungraded: resources-dynamic-marker (a harness cannot tell whether content is generated), enumeration-scheme-uniform, skills-get-answers-unlisted (no way to find an unlisted URI on an arbitrary server), read-is-not-a-load, and the MAY rows.

Risk / blast radius

  • Affects: the two skills server scenarios gain checks, and the client suite gains four scenarios. No runner or shared-helper change.
  • Tests covering this: 23 new vitest cases, and the full suite passes (672).
  • Second implementation: the server checks run against mcpkit examples/skills at 51 pass / 0 fail / 0 warn / 0 skip. The client scenarios pass against mcpkit's client. One of them found a real mcpkit bug: a single invalid entry failed the whole skills/list decode (skills: grade 13 more SEP-2640 rows, and keep a listing with one invalid entry loadable panyam/mcpkit#1728).
  • Draft because: the client scenarios have run against one real client so far. I'd like a second one, such as the Go SDK, before marking this ready.

Server scenarios gain six checks. enumeration.ts grades nested-skill
completeness, a flat listing, -32602 for an unserved skill file, and an
informative error message. directory.ts grades ttlMs and cacheScope on
the directory result, and resources: [] for an empty directory.

A new client file, hosting.ts, adds four scenarios with seven checks:
loading a skill by URI after an empty listing, refusing invalid entries,
refusing a directory child the entry does not list, and keeping two
servers' skills apart. The two-origins scenario starts a second mock
server and hands its URL to the client in MCP_CONFORMANCE_CONTEXT, the
way the auth scenarios hand over an authorization server.

Every new check has a negative control. The sep-2640.yaml header now
counts 59 of 89 rows as emitted, up from 46.
@pkg-pr-new

pkg-pr-new Bot commented Nov 16, 2026

Copy link
Copy Markdown

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@590

commit: f3768ba

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant