feat(mcp): authenticated read-only Streamable HTTP preview (#518)

## Summary
Adds an opt-in Streamable HTTP endpoint using the existing RMCP SDK and
Sem handlers. Stdio remains unchanged.

- `sem mcp --http 127.0.0.1:8765` (also standalone `sem-mcp --http`)
- Explicit SEM_REPO, mandatory random bearer token, loopback-only bind,
exact Host check, rejects browser Origin requests
- Only sem_find and sem_grep advertised AND callable; no validation
commands, cloud tools, or Weave writes
- Shared repository caches, stateless requests with independent context
ledgers
- 1 MiB body limit and eight concurrent HTTP requests; relative-path and
explicit symlink escape checks
- Source-build documentation, release limitation, and smoke test in
existing Linux/macOS CI

## Validation
- Built sem-cli and sem-mcp locally
- 100 sem-mcp library tests passed
- 12 existing MCP protocol tests passed
- 7 live HTTP smoke tests passed (handshake, reads, lookup/grep,
hidden-tool denial, paths, auth/Host/Origin/body size, nonloopback
rejection)
- Actual LangChain MCP adapter 0.2.1/MCP 1.25 and Aser/FastMCP 4.1
clients retrieved a function and caller over HTTP

## Deliberate boundaries
This is a trusted-client local/SSH-tunnel preview, NOT public hosting,
OAuth, a multi-tenant sandbox, or a remote editing service. It does not
make arbitrary checkouts safe. Published 0.27.0 does not include this
feature. No speedup claim. Downstream examples will be drafts pending
this transport landing.

AI-assisted implementation; local toolchain emitted existing LLVM
debug-stripping warnings but builds and tests completed successfully.

Author: rs545837

.github/workflows/shared-mcp.yml

@@ -19,4 +19,5 @@ jobs:
           workspaces: crates
       - run: cargo build --manifest-path crates/Cargo.toml -p sem-mcp -p sem-cli
       - run: cargo test --manifest-path crates/Cargo.toml -p sem-mcp --lib --tests
+      - run: python3 crates/sem-mcp/tests/http_smoke.py crates/target/debug/sem-mcp
       - run: python3 benchmarks/shared-mcp/run.py --binary crates/target/debug/sem-mcp

CHANGELOG.md

@@ -4,6 +4,9 @@ All notable changes to sem are documented in this file.
 
 ## [Unreleased]
 
+### Added
+- Preview authenticated, loopback-only Streamable HTTP MCP via `sem mcp --http`, exposing read-only `sem_find` and `sem_grep` with shared repository caches. See `docs/mcp-http.md` for source-build instructions and deployment limits.
+
 ### Added
 
 - **`sem_find` reads whole files, so an agent never needs Read, `cat` or `sed` to look at code or config.** With mode `context` and no query, it reads `files: [...]` (or the file in `in`) in one call, in parallel, numbered like `cat -n`, with an optional `lines` range. A file the session already read answers in one line when it is unchanged, and with a line diff against the version the agent saw when it changed; `fresh: true` resends it whole.

README.md

@@ -602,6 +602,8 @@ This means sem detects renames and moves, not just additions and deletions. Stru
 
 ## Use with AI agents (MCP)
 
+Need a URL-based client? The [Streamable HTTP preview](docs/mcp-http.md) adds authenticated, local read-only access; stdio remains the default.
+
 On macOS and Linux, clients in the same checkout share a warm repository daemon.
 Run `sem mcp --status` to check it. See the [shared runtime contract and reproducible benchmark](docs/shared-mcp.md)
 for session isolation, fallback behavior, and current platform limits.

crates/Cargo.lock

@@ -917,6 +917,10 @@ enum Commands {
     /// Run sem as an MCP server for agents (stdin/stdout): find, grep, impact, check, certify, diff, graph, history
     #[command(display_order = 11)]
     Mcp {
+        /// Serve read-only discovery over authenticated Streamable HTTP on loopback.
+        /// Requires SEM_REPO and SEM_MCP_HTTP_TOKEN (at least 32 characters).
+        #[arg(long, conflicts_with_all = ["status", "resident", "review"])]
+        http: Option<std::net::SocketAddr>,
         /// Check shared MCP daemon health without starting it (JSON). Example: sem mcp --status
         #[arg(long, conflicts_with = "resident")]
         status: bool,
@@ -1916,7 +1920,7 @@ fn sem_main() {
             alias::note("stats", "config stats", false);
             commands::stats::run();
         }
-        Some(Commands::Mcp { resident, status, review }) => {
+        Some(Commands::Mcp { resident, status, review, http }) => {
             if status {
                 println!("{}", sem_mcp::shared_status());
                 return;
@@ -1925,7 +1929,9 @@ fn sem_main() {
                 // No-op: see the `resident` field's doc comment above.
                 return;
             }
-            let result = if review { sem_mcp::run_review() } else { sem_mcp::run() };
+            let result = if let Some(address) = http {
+                sem_mcp::http::run(address)
+            } else if review { sem_mcp::run_review() } else { sem_mcp::run() };
             if let Err(e) = result {
                 eprintln!("{} {}", "error:".red().bold(), e);
                 std::process::exit(1);

crates/sem-cli/src/main.rs

@@ -18,7 +18,10 @@ sem-core = { path = "../sem-core", version = "0.27.0", features = ["disk-cache"]
 sem-cloud-client = { path = "../sem-cloud-client", version = "0.27.0" }
 git2 = "0.20"
 lru = "0.16"
-rmcp = { version = "1", features = ["server", "transport-io"] }
+rmcp = { version = "1", features = ["server", "transport-io", "transport-streamable-http-server"] }
+axum = "0.8"
+subtle = "2"
+tower-http = { version = "0.6", features = ["limit"] }
 tokio = { version = "1", features = ["full"] }
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
@@ -34,7 +37,6 @@ ureq = { version = "2", features = ["json"] }
 
 [dev-dependencies]
 tempfile = "3.24.0"
-axum = "0.8"
 # The cache test module (src/cache.rs) asserts against raw `rusqlite` types
 # directly (schema/row-level checks predating DiskCache move to
 # sem-core); production code no longer touches `rusqlite` itself, so this is