Skip to content

Commit 5932521

Browse files
committed
Pin the conformance CLI to the alpha dist-tag
## Motivation and Context The npm `latest` dist-tag of `@modelcontextprotocol/conformance` points at 0.1.16, the old 0.1 line, while the 2026-07-28 requirement revisions and the frozen-requirements framework live in the 0.2.0 alpha line (`alpha` currently resolves 0.2.0-alpha.11). The runners invoked the bare package name, so `rake conformance` and the conformance CI job resolve whatever `latest` points at: a fresh machine silently downgrades to the old suite while a warm npx cache may keep serving a newer one, and the suite actually in use depends on the environment. Pinning the `alpha` dist-tag puts every run on the line the frozen conformance requirements are anchored to (0.2.0-alpha.10, published 2026-07-27). Running the real alpha suite surfaced the actual client-leg state, so the expected-failures baseline is updated to match: `initialize` now passes (the harness accepts 2026-07-28, fulfilling the TODO on the old entry), and the scenarios pending client-side work are recorded with their reasons: the 2026-07-28 stateless lifecycle and the SEP-2322 MRTR driver, SEP-2243 custom-header mirroring, client-side JSON Schema preservation, the DPoP and WIF authorization extensions, and one `auth/authorization-server-migration` check added to the suite after the alpha.10 anchor. The manual `npx` examples in `conformance/README.md` carry the same pin so local reproduction runs the same suite as the rake task. ## How Has This Been Tested? `npx --yes @modelcontextprotocol/conformance@alpha --version` resolves 0.2.0-alpha.11, and `bundle exec rake conformance` passes the baseline check on both legs: the server leg reports 73 passed with 0 failed and no expected failures, and the client leg fails exactly the scenarios the baseline records. ## Breaking Changes None. The change is confined to the conformance tooling; the shipped gem is untouched.
1 parent 05c3585 commit 5932521

4 files changed

Lines changed: 23 additions & 10 deletions

File tree

‎conformance/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,10 @@ bundle exec rake conformance_server PORT=3000
4747
bundle exec rake conformance_server
4848

4949
# Terminal 2: run all scenarios
50-
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp
50+
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp
5151

5252
# Terminal 2: run a single scenario
53-
npx @modelcontextprotocol/conformance server --url http://localhost:9292/mcp --scenario ping
53+
npx @modelcontextprotocol/conformance@alpha server --url http://localhost:9292/mcp --scenario ping
5454
```
5555

5656
Keeps the server alive between test runs, which avoids the startup overhead when iterating

‎conformance/client_runner.rb‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# frozen_string_literal: true
22

3-
# Runs `npx @modelcontextprotocol/conformance client` against the conformance client script.
3+
# Runs `npx @modelcontextprotocol/conformance@alpha client` against the conformance client script.
44
require "English"
55

66
module Conformance
@@ -29,7 +29,7 @@ def build_command
2929
npx_command = [
3030
"npx",
3131
"--yes",
32-
"@modelcontextprotocol/conformance",
32+
"@modelcontextprotocol/conformance@alpha",
3333
"client",
3434
"--command",
3535
"bundle exec ruby #{client_script}",

‎conformance/expected_failures.yml‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,19 @@
11
server: []
22
client:
3-
# TODO: The conformance framework's client-initialization check pins the expected requested version to 2025-11-25,
4-
# while this SDK's latest protocol version is 2026-07-28. Remove once the @modelcontextprotocol/conformance package
5-
# accepts 2026-07-28.
6-
- initialize
3+
# Pending client-side support for the 2026-07-28 stateless lifecycle: the conformance client
4+
# still connects through the legacy handshake and has no SEP-2322 MRTR driver yet.
5+
- request-metadata
6+
- sep-2322-client-request-state
7+
# SEP-2243 client-side custom-header mirroring and invalid-tool header gating
8+
# are not implemented in `MCP::Client::HTTP` yet.
9+
- http-custom-headers
10+
- http-invalid-tool-headers
11+
# Client-side JSON Schema preservation gaps: `$ref` values must reach the caller unresolved.
12+
- json-schema-ref-no-deref
13+
- json-schema-2020-12-preservation
14+
# Unimplemented authorization extensions.
15+
- auth/dpop
16+
- auth/dpop-nonce
17+
- auth/wif-jwt-bearer
18+
# Fails one check added to the suite after the 0.2.0-alpha.10 requirements anchor.
19+
- auth/authorization-server-migration

‎conformance/server_runner.rb‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# frozen_string_literal: true
22

3-
# Starts the conformance server and runs `npx @modelcontextprotocol/conformance` against it.
3+
# Starts the conformance server and runs `npx @modelcontextprotocol/conformance@alpha` against it.
44
require "English"
55
require "net/http"
66
require_relative "server"
@@ -31,7 +31,7 @@ def run
3131
def build_command
3232
expected_failures_yml = File.expand_path("expected_failures.yml", __dir__)
3333

34-
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance", "server", "--url", "http://localhost:#{@port}/mcp"]
34+
npx_command = ["npx", "--yes", "@modelcontextprotocol/conformance@alpha", "server", "--url", "http://localhost:#{@port}/mcp"]
3535
npx_command += ["--scenario", @scenario] if @scenario
3636
npx_command += ["--spec-version", @spec_version] if @spec_version
3737
npx_command += ["--verbose"] if @verbose

0 commit comments

Comments
 (0)