feat(middleware): Serve production build URLs to the evals pin header (#20225)

## DESCRIBE YOUR PR

Production build URLs (`sentry-docs-<hash>.sentry.dev`) now serve their
own pages to requests that have an `x-sentry-evals-pin` header, instead
of 308ing them to docs.sentry.io. Everyone else is still redirected as
in #19033, and these responses still get `X-Robots-Tag: noindex`.

[sentry-for-ai-evals](https://github.com/getsentry/sentry-for-ai-evals)
compares a docs PR's preview with the docs at the PR's merge base, and
that commit's production build is the only copy of those. The redirect
sends the evals to the live docs instead, so a PR that's behind master
gets everything master gained since counted as its own change. The
evals' docs proxy sends the header, and #20214 points the base at the
merge base's build. Like the redirect, the header isn't a security
boundary: previews are already public, and an old build only has docs
that were once live.

Only builds made after this merges check for the header, so older merge
bases keep reading the live docs.

Refs
[VDY-307](https://linear.app/getsentry/issue/VDY-307/onboarding-evals-ai-evals-label-workflow-in-sentry-docs)

## IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs
to go live.
Select exactly one option. For deadlines, replace `YYYY-MM-DD` with the
due date. You can update this information later by editing the PR
description.

- [ ] Urgent deadline (GA date, etc.): YYYY-MM-DD
- [ ] Other deadline: YYYY-MM-DD
- [x] No deadline: Not urgent, can wait up to 1 week+

## SLA

- Teamwork makes the dream work, so please add a reviewer to your PRs.
- Please give the docs team up to 1 week to review your PR unless you've
supplied a deadline.

Thanks in advance for your help!

## PRE-MERGE CHECKLIST

_Make sure you've checked the following before merging your changes:_

- [ ] Checked Vercel preview for correctness, including links
- [ ] PR was reviewed and approved by any necessary SMEs (subject matter
experts)
- [ ] PR was reviewed and approved by a member of the [Sentry docs
team](https://github.com/orgs/getsentry/teams/docs)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

Author: jaydgoss

middleware.test.ts

@@ -113,8 +113,12 @@ describe('production build-url redirect to canonical (Deployment Protection off)
     vi.unstubAllEnvs();
   });
 
-  function makeHostRequest(url: string, method = 'GET'): NextRequest {
-    return new NextRequest(new URL(url), {method});
+  function makeHostRequest(
+    url: string,
+    method = 'GET',
+    headers?: HeadersInit
+  ): NextRequest {
+    return new NextRequest(new URL(url), {method, headers});
   }
 
   it('redirects a non-canonical production host to the canonical host', async () => {
@@ -192,6 +196,17 @@ describe('production build-url redirect to canonical (Deployment Protection off)
     );
     expect(res.status).not.toBe(308);
   });
+
+  it('serves a production build URL to the evals pin header', async () => {
+    const {middleware} = await importMiddleware({});
+    vi.stubEnv('VERCEL_ENV', 'production');
+    const res = middleware(
+      makeHostRequest('https://sentry-docs-abc123.vercel.app/llms.txt', 'GET', {
+        'x-sentry-evals-pin': '1',
+      })
+    );
+    expect(res.status).not.toBe(308);
+  });
 });
 
 describe('canonical Link header on .md responses', () => {

middleware.ts

@@ -19,6 +19,11 @@ const BASE_URL = isDeveloperDocs
 
 const CANONICAL_HOST = new URL(BASE_URL).hostname;
 
+// sentry-for-ai-evals compares a PR's docs with the docs at its merge base, which it reads
+// from that commit's production build URL. It sends this header so the build serves its
+// own pages instead of redirecting to the canonical host.
+const EVALS_PIN_HEADER = 'x-sentry-evals-pin';
+
 // Production domains whose content should be indexable by search engines.
 // All other hostnames (Vercel preview/deployment URLs, old production deployments)
 // get X-Robots-Tag: noindex to prevent search engines from indexing stale content.
@@ -73,7 +78,10 @@ function redirectProductionBuildUrlToCanonical(
   ) {
     return null;
   }
-  if (request.nextUrl.hostname === CANONICAL_HOST) {
+  if (
+    request.nextUrl.hostname === CANONICAL_HOST ||
+    request.headers.has(EVALS_PIN_HEADER)
+  ) {
     return null;
   }
   const url = request.nextUrl.clone();