fix(docs): curate LLM exports and correct authentication guidance (#1734)

## Problem and change

The documentation LLM index advertised 71 unpublished pages that
returned 404, while the full export included internal research, plans
and historical investigations. Share the HTML publication
allowlist/exclusions with the LLM plugin so both exports contain only
published docs. The resulting index contains 129 links, all matched to
Docusaurus metadata and generated HTML.

Correct the MCP authentication reference to distinguish optional local
HTTP auth, compatibility admin access for anonymous callers, enforced
auth, Server edition and OS-authenticated local transports. Replace the
homepage's overclaiming diagram with an accessible SVG stating the
limits of protection. Add descriptions to seven category pages, exclude
noindexed search from the sitemap, and publish robots.txt with the
sitemap location.

## Validation

- `npm ci --no-audit --no-fund`
- `npm run build` — production build and automatic postbuild validation
passed.
- New verifier reproduced the original defect: `LLM export links to
unpublished page: https://docs.mcpproxy.app/docs/E2E_TESTING`.
- Fixed output: `Verified 129 published LLM links, full-export
regression markers, category metadata, sitemap and robots.txt.`
- Mutation checks rejected an injected unpublished URL and internal
workflow marker; generated files restored and verifier passed again.
- Browser inspection confirmed the SVG and authentication table render
correctly.
- Independent fresh-context review: `VERDICT: PASS` after correcting the
anonymous-admin compatibility wording. No remaining actionable findings.

The postbuild gate runs through the existing PR, main deployment and
release `npm run build` paths. Runtime behavior is unchanged. Broader
navigation and per-page Markdown work remain outside this PR.

## Deployed preview

[Cloudflare preview](https://68e931e9.mcpproxy-docs.pages.dev/) verified
live: all 129 exported routes load; the full export excludes the known
internal-note regression markers; the auth matrix, replacement SVG,
sitemap and robots declaration are present. Full export is approximately
1.43 MB, down from 3.04 MB in the audited production snapshot.

The publication arrays remain literal in `docusaurus.config.js` for the
existing release-gate parser, while both builders reference that same
object. Reproduced the three parser failures from the first CI run,
fixed compatibility, and passed `go test ./cmd/release-gate -count=1`, a
fresh production docs build, and independent follow-up review (`VERDICT:
PASS`).

Author: Dumbris

docs/api/mcp-protocol.md

@@ -34,7 +34,23 @@ Each routing mode has a dedicated endpoint that is always available regardless o
 
 See [Routing Modes](../features/routing-modes.md) for details on each mode.
 
-**Note:** MCP endpoints do not require API key authentication for client compatibility.
+### Authentication
+
+Authentication depends on the deployment and transport:
+
+| Surface | Authentication |
+| --- | --- |
+| Local MCP over HTTP, default configuration | Authentication is optional (`require_mcp_auth: false`). Anonymous callers retain admin-level tool and management access for compatibility, subject to enabled tools and configuration gates; they cannot use the opt-in raw secret-header reveal. |
+| MCP over HTTP with `require_mcp_auth: true` | Requires a valid credential, such as an admin API key or a scoped agent token. Enable this before exposing MCP beyond localhost. |
+| Server edition with `server_edition.enabled: true` | Forces MCP authentication on, even if `require_mcp_auth` is explicitly false. |
+| REST API over HTTP | Requires authentication independently of `require_mcp_auth`. |
+| Local Unix socket / Windows named pipe | Uses OS-level access control; trusted local connections do not need an API key. |
+
+See [configuration defaults](/configuration/config-file#server-settings),
+[agent tokens and MCP authentication](/features/agent-tokens#enforcing-authentication-on-mcp),
+[REST authentication](/api/rest-api#authentication), and
+[Server edition authentication](/development/server-edition-multiuser-auth)
+for credential formats and access rules.
 
 ## Built-in Tools
 

docs/intro.md

@@ -12,7 +12,7 @@ keywords: [mcp, proxy, ai agents, tool discovery, security, docker]
 
 **A smart proxy for AI agents using the Model Context Protocol (MCP)**
 
-![MCPProxy Architecture](/img/mcpproxy-architecture.jpeg)
+![AI clients connect to MCPProxy, which discovers tools across upstream MCP servers. Review and isolation controls reduce risk but do not guarantee safety.](/img/mcpproxy-architecture.svg)
 
 ## What is MCPProxy?
 

website/docusaurus.config.js

@@ -3,6 +3,38 @@
 
 const {themes: prismThemes} = require('prism-react-renderer');
 
+// Shared publication boundary for HTML pages and both LLM exports.
+// Keep internal research, plans and QA notes outside these patterns.
+// Keep these literal lists here: cmd/release-gate also reads them to guard
+// claims and links across every published page.
+const publishedDocs = {
+  include: [
+    'intro.md',
+    'getting-started/**/*.{md,mdx}',
+    'configuration/**/*.{md,mdx}',
+    'cli/**/*.{md,mdx}',
+    'api/**/*.{md,mdx}',
+    'web-ui/**/*.{md,mdx}',
+    'features/**/*.{md,mdx}',
+    'code_execution/**/*.{md,mdx}',
+    'operations/**/*.{md,mdx}',
+    'errors/**/*.{md,mdx}',
+    'development/**/*.{md,mdx}',
+    'contributing.md',
+    // Standalone references with no structured counterpart
+    'cli-client-mode.md',
+    'cli-output-formatting.md',
+    'logging.md',
+    'prerelease-builds.md',
+    'registries.md',
+    'socket-communication.md',
+  ],
+  exclude: [
+    'development/sandbox-snap-docker-harness.md',
+    'development/sandbox-spike-mcp-34.md',
+  ],
+};
+
 /** @type {import('@docusaurus/types').Config} */
 // Version shown in the navbar badge. Supplied by CI: docs.yml derives it from
 // the latest stable release tag reachable from main, and release.yml's
@@ -53,35 +85,11 @@ const config = {
           routeBasePath: '/', // docs at root
           sidebarPath: './sidebars.js',
           editUrl: 'https://github.com/smart-mcp-proxy/mcpproxy-go/edit/main/',
-          // Only include structured documentation pages
-          include: [
-            'intro.md',
-            'getting-started/**/*.{md,mdx}',
-            'configuration/**/*.{md,mdx}',
-            'cli/**/*.{md,mdx}',
-            'api/**/*.{md,mdx}',
-            'web-ui/**/*.{md,mdx}',
-            'features/**/*.{md,mdx}',
-            'code_execution/**/*.{md,mdx}',
-            'operations/**/*.{md,mdx}',
-            'errors/**/*.{md,mdx}',
-            'development/**/*.{md,mdx}',
-            'contributing.md',
-            // Standalone references with no structured counterpart
-            'cli-client-mode.md',
-            'cli-output-formatting.md',
-            'logging.md',
-            'prerelease-builds.md',
-            'registries.md',
-            'socket-communication.md',
-          ],
-          // Internal working documents that live in the repo but are not published
-          exclude: [
-            'development/sandbox-snap-docker-harness.md',
-            'development/sandbox-spike-mcp-34.md',
-          ],
+          include: publishedDocs.include,
+          exclude: publishedDocs.exclude,
         },
         blog: false,
+        sitemap: { ignorePatterns: ['/search'] },
         theme: {
           customCss: './src/css/custom.css',
         },
@@ -97,16 +105,10 @@ const config = {
         generateLLMsFullTxt: true,
         excludeImports: true,
         removeDuplicateHeadings: true,
-        includeOrder: [
-          'getting-started/*',
-          'configuration/*',
-          'cli/*',
-          'api/*',
-          'web-ui/*',
-          'features/*',
-          'code_execution/*',
-          'operations/*',
-        ],
+        docsDir: [{ path: 'docs', routeBasePath: '/' }],
+        includeOrder: publishedDocs.include,
+        includeUnmatchedLast: false,
+        ignoreFiles: publishedDocs.exclude,
       },
     ],
   ],

website/package.json

@@ -11,7 +11,9 @@
     "clear": "docusaurus clear",
     "serve": "docusaurus serve",
     "write-translations": "docusaurus write-translations",
-    "write-heading-ids": "docusaurus write-heading-ids"
+    "write-heading-ids": "docusaurus write-heading-ids",
+    "postbuild": "node scripts/verify-docs-build.js",
+    "verify:build": "node scripts/verify-docs-build.js"
   },
   "dependencies": {
     "@docusaurus/core": "^3.10.2",

website/scripts/verify-docs-build.js

@@ -0,0 +1,49 @@
+#!/usr/bin/env node
+// Verify the actual generated artifacts, not just the source glob settings.
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+
+const siteDir = path.resolve(__dirname, '..');
+const buildDir = path.join(siteDir, 'build');
+const metadataDir = path.join(siteDir, '.docusaurus/docusaurus-plugin-content-docs/default');
+const normalize = value => new URL(value, 'https://docs.mcpproxy.app').pathname.replace(/\/$/, '') || '/';
+const read = file => fs.readFileSync(path.join(buildDir, file), 'utf8');
+const metadata = fs.readdirSync(metadataDir)
+  .filter(file => file.endsWith('.json') && !file.startsWith('__'))
+  .map(file => JSON.parse(fs.readFileSync(path.join(metadataDir, file), 'utf8')))
+  .filter(doc => doc.source && doc.permalink && !doc.draft && !doc.unlisted);
+assert(metadata.length > 0, 'No published document metadata found');
+const published = new Set(metadata.map(doc => normalize(doc.permalink)));
+const index = read('llms.txt');
+const links = [...index.matchAll(/^- \[[^\n]*\]\((https:\/\/docs\.mcpproxy\.app[^)\s]*)\)/gm)].map(match => match[1]);
+assert(links.length > 0, 'LLM index is empty');
+const exported = new Set();
+for (const link of links) {
+  const route = normalize(link);
+  assert(published.has(route), `LLM export links to unpublished page: ${link}`);
+  assert(!exported.has(route), `Duplicate LLM export route: ${link}`);
+  exported.add(route);
+  const relative = route.replace(/^\//, '');
+  assert([path.join(buildDir, relative, 'index.html'), path.join(buildDir, `${relative}.html`)]
+    .some(file => fs.existsSync(file)), `LLM export has no built HTML page: ${link}`);
+}
+assert.deepEqual([...exported].sort(), [...published].sort(), 'LLM index must cover every published document');
+const full = read('llms-full.txt');
+assert(full.length > index.length, 'Full LLM export is empty or incomplete');
+// Regressions from the original export: excluded docs had no HTML routes but
+// their bodies (including agent workflow instructions) appeared in llms-full.
+for (const marker of [
+  'CRITICAL SECURITY VULNERABILITY: Skipped API Key Tests',
+  'REQUIRED SUB-SKILL:',
+  'TPA Scanner: Inline Findings & Background Scanning — Authoritative Design',
+]) {
+  assert(!full.includes(marker), `Unpublished content in full LLM export: ${marker}`);
+}
+assert(!/<loc>https:\/\/docs\.mcpproxy\.app\/search\/?<\/loc>/.test(read('sitemap.xml')), 'Noindexed search page is in sitemap');
+assert(read('robots.txt').includes('Sitemap: https://docs.mcpproxy.app/sitemap.xml'), 'robots.txt is missing sitemap declaration');
+for (const category of ['getting-started', 'configuration', 'cli', 'api', 'web-ui', 'features', 'operations']) {
+  const html = fs.existsSync(path.join(buildDir, category, 'index.html')) ? read(`${category}/index.html`) : read(`${category}.html`);
+  assert(/<meta[^>]*name="description"[^>]*content="[^"]+"/.test(html), `Missing category description: ${category}`);
+}
+console.log(`Verified ${exported.size} published LLM links, full-export regression markers, category metadata, sitemap and robots.txt.`);

website/sidebars.js

@@ -24,7 +24,7 @@ const sidebars = {
       // reads _category_.json for AUTOGENERATED sidebars. Without this the
       // /getting-started URL 404s, which is where the marketing site's
       // "Getting Started" card used to land.
-      link: { type: 'generated-index', slug: '/getting-started' },
+      link: { type: 'generated-index', slug: '/getting-started', description: 'Install MCPProxy and make your first successful MCP tool call.' },
       items: [
         'getting-started/installation',
         'getting-started/quick-start',
@@ -34,7 +34,7 @@ const sidebars = {
       type: 'category',
       label: 'Configuration',
       // See the note on Getting Started above — same reason.
-      link: { type: 'generated-index', slug: '/configuration' },
+      link: { type: 'generated-index', slug: '/configuration', description: 'Configure MCPProxy, upstream servers, environment variables and sensitive data detection.' },
       items: [
         'configuration/config-file',
         'configuration/upstream-servers',
@@ -45,7 +45,7 @@ const sidebars = {
     {
       type: 'category',
       label: 'CLI',
-      link: { type: 'generated-index', slug: '/cli' },
+      link: { type: 'generated-index', slug: '/cli', description: 'Manage MCPProxy servers, profiles, security reviews and activity from the command line.' },
       items: [
         'cli/command-reference',
         'cli/management-commands',
@@ -66,7 +66,7 @@ const sidebars = {
     {
       type: 'category',
       label: 'API',
-      link: { type: 'generated-index', slug: '/api' },
+      link: { type: 'generated-index', slug: '/api', description: 'Integrate with the MCPProxy REST API and MCP protocol endpoints.' },
       items: [
         'api/rest-api',
         'api/mcp-protocol',
@@ -75,7 +75,7 @@ const sidebars = {
     {
       type: 'category',
       label: 'Web UI',
-      link: { type: 'generated-index', slug: '/web-ui' },
+      link: { type: 'generated-index', slug: '/web-ui', description: 'Use the MCPProxy dashboard to manage servers, settings and activity.' },
       items: [
         'web-ui/dashboard',
         'web-ui/server-detail',
@@ -86,7 +86,7 @@ const sidebars = {
     {
       type: 'category',
       label: 'Features',
-      link: { type: 'generated-index', slug: '/features' },
+      link: { type: 'generated-index', slug: '/features', description: 'Explore tool discovery, routing, client connections and security controls.' },
       items: [
         'features/routing-modes',
         'features/schema-deferred-direct-mode',
@@ -156,7 +156,7 @@ const sidebars = {
     {
       type: 'category',
       label: 'Operations',
-      link: { type: 'generated-index', slug: '/operations' },
+      link: { type: 'generated-index', slug: '/operations', description: 'Deploy, monitor and maintain MCPProxy locally or for a team.' },
       items: [
         'operations/reverse-proxy',
         'operations/deploying-for-a-team',

website/static/img/mcpproxy-architecture.svg

@@ -0,0 +1,4 @@
+User-agent: *
+Allow: /
+
+Sitemap: https://docs.mcpproxy.app/sitemap.xml