Allow pages to be served in markdown for AI Agents

[gusfune]

Goals

1. Alternatively serve text/markdown for agents retrieve content quickly
2. Allow this via a similar parameter such as generateMetadata but tentatively call it generateStaticMarkdownContent
3. Service those pages when the headers request such via accept or content-type requests
4. Should be compatible with app router in page.tsx (not on page router)

Non-Goals

1. Convert the content to markdown automatically, this is not in the scope
2. Implement via a plugin as it's a big change in the routing

Background

With LLMs parsing a lot of content of the pages this is becoming more relevant. Cloudflare announced recently this feature on sites served on it, but ideally full control would happen at the application: https://blog.cloudflare.com/markdown-for-agents/

Proposal

• Modify app-render.tsx to allow support for text/markdown in header
• Create a new function generateStaticTextContent similar to generateStaticParams that will allow control on what to be served. If that function exists for a page, that page will support the markdown alternative route

[icyJoseph]

Just to inform people. The framework is flexible enough that you can do a rewrite based on incoming headers, namely Accept and use that to rewrite to a route that serves markdown.

For example in your Next config file:

        {
          source: '/docs/:slug',
          destination: '/docs/md/:slug',
          has: [
            {
              type: 'header',
              key: 'accept',
              value: '(.*)text/markdown(.*)',
            },
          ],
        },

And then you'd have, apps/docs/md/[...slug]/route.ts - Where you generate the markdown.

export async function generateStaticParams() {
  return [/* the set of slugs you wanna prerender */];
}

export async function GET(
  _: Request,
  { params }: { params: Promise<{ slug: string[] }> },
) {
  const { slug } = await params;

  return new Response(/* your content */, {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      Vary: 'Accept',
    },
  });
}

Unlike the llms.txt proposal, which is kind of a simpler API and as time goes on should standardize more, I am not sure about this proposal being the right level of abstraction. I have a few more questions/considerations, but let's see what others have to say.

[rilexus]

I was looking for something like this. Thanks!

[KKonstantinov]

The rewrite approach described above works well if you want full control over the markdown output. For cases where you want automatic conversion, I built a Next.js middleware for this - inspired by the Cloudflare feature mentioned in the original post, but at the application layer:

npm install @markdown-for-agents/nextjs
// middleware.ts
import { withMarkdown } from '@markdown-for-agents/nextjs';

export default withMarkdown();

Handles content negotiation via Accept: text/markdown, strips non-content elements, sets caching headers. Details in the repo: https://github.com/KKonstantinov/markdown-for-agents

[KKonstantinov]

Thanks for the feedback and for trying it out! You're right - the original example had issues. Have since fixed the docs, the types, and added an examples/nextjs with integration tests.

On the double fetch - Next.js proxy runs before page rendering with no access to the response body, so there's no way around it at the application layer. In practice it's localhost-to-localhost (or edge-to-edge on Vercel), and static/ISR pages are cache hits, so overhead should be minimal

Totally agree this is a rich problem space. A platform-level solution like your reverse proxy approach or what Cloudflare is doing avoids the double fetch entirely. This package is meant as an application-layer option for when you don't have that infrastructure, or want control over extraction, deduplication, and content signals.

There are certain items that would be hard to tackle on the infrastructure layer though:

• Not all infra's will implement this, leaving out some users
• The ones that do, I do not think they will provide rich customization opportunities (e.g. they will map HTML to Markdown on a 1:1 basis) - however there are elements such as Call To Actions (CTAs) and others like these, for which agents do not necessarily care
• An application-layer library could provide fine-tuning capabilities on the conversion

Cool project with mdx2md - I'll check it out, thanks for sharing!

[icyJoseph]

In practice ... (or edge-to-edge on Vercel), and static/ISR pages are cache hits, so overhead should be minimal

This is unfortunately not true, and often, the reason why some apps are slow. The HTTP request does a full roundtrip - of course, app design is full of trade-offs, so some apps do need to make an extra call there, but ideally, it should be your very last resort.

There's also the caveat of, whether or not, some, if not all content can have its markdown complement made at build time.

There are a lot of hints at this being either a platform game, or a rendering engine one. Should React expose a way to emit markdown chunks instead? Next.js is the rendering delivery engine, React's the one who knows your component tree deeply.

[KKonstantinov]

That's a good point, thanks - my assumption was wrong on this.

Its worth doing a performance test on the nextjs adapter and see how much the overhead is.

For the time being though, I would like to understand whether having the capability on nextjs (even with the overhead) outweighs not having the capability at all - doing some performance testing will help - could add that measurement to READMEs.

It would be great if there was a more optimal way to do in on the application layer for NextJs.

[icyJoseph]

Now that's a good counter. Benchmark it!

Yeah as said, the one who knows about your app's HTML is React. Anything else, even generateMarkdown or which ever way, is ad-hoc, and made to patch the rendered result.

I could also ask, well, what if I want to serve with ANSI codes and stuff, for humans visiting my site on a terminal. Again, React know's what's up in my app, it could serve different content, not just HTML.

User land solutions will have to do for now though.

[KKonstantinov]

Here are the metrics; For what it's worth, I'm using on my website & a 300ms tradeoff against not having it is worth it (for me). Caching plays a role here, were subsequent calls are much better, find below.

PS. We should probably stop hijacking this thread. :) DM me if you like - keen on discussing, and thank you for looking into this.

Page	Tokens	Vercel cache	Proxy fetch	Conversion	Total
Landing page	~500	HIT	31ms	1.4ms	32ms
About page (cold)	~1,000	PRERENDER	325ms	1.7ms	327ms
About page (warm)	~1,000	HIT	43ms	1.5ms	45ms
Long article (cold)	~3,200	PRERENDER	443ms	76ms	519ms
Long article (warm)	~3,200	HIT	36ms	21ms	57ms