Making agent-friendly pages with content negotiation

Agents fetch web pages to answer questions, write code, and complete tasks. When an agent requests a page, it gets everything your browser gets, including navigation menus, stylesheets, JavaScript bundles, tracking scripts, and footer links, when all it needs is the structured text on the page. That extra markup confuses the agent, consumes its context window, and makes every request more expensive.

Link to headingWhat is content negotiation

What agents need is a way to request just the text content of a page, without the browser-specific markup. Content negotiation solves this. It's a standard HTTP mechanism where the client specifies its preferred format via the Accept header, and the server returns the matching representation. Many agents already send Accept: text/markdown when fetching pages, and a server that supports content negotiation can return clean, structured text from the same URL that serves HTML to a browser.

We've updated many of our pages, including our blog and changelog, to support content negotiation. This post walks through how it works, how we implemented it in Next.js, and how to add markdown sitemaps so agents can discover your content.

Link to headingHow agents request markdown

When an agent fetches a page, it includes an Accept header with its format preferences:

Accept: text/markdown, text/html, */*

By listing text/markdown first, the agent signals that markdown is preferred over HTML when available. This works better than hosting separate .md URLs because content negotiation requires no site-specific knowledge. Any agent that sends the right header gets markdown automatically, from any site that supports it.

Try it yourself:

curl https://vercel.com/blog/self-driving-infrastructure -H "accept: text/markdown"

Link to headingImplementing content negotiation in Next.js

The implementation has two parts, a rewrite rule in next.config.ts that detects the header and a route handler that returns markdown.

The rewrite checks the Accept header on every incoming request. When it contains text/markdown, the request gets routed to a dedicated markdown endpoint instead of the default HTML page:

1
import type { NextConfig } from 'next';
2

3
const nextConfig: NextConfig = {
4
  async rewrites() {
5
    function markdownRewrite(prefix) {
6
      return {
7
        source: `${prefix}/:path*`,
8
        has: [
9
          {
10
            type: 'header',
11
            key: 'accept',
12
            value: '(.*)text/markdown(.*)',
13
          },
14
        ],
15
        destination: `${prefix}/md/:path*`,
16
      };
17
    }
18
    return {
19
      beforeFiles: [markdownRewrite('/blog'), markdownRewrite('/changelog')],
20
    };
21
  },
22
};
23

24
export default nextConfig;

The route handler serves the markdown. Our blog content lives in our CMS as rich text, so the route handler converts it to markdown on the fly. If your content is already authored in markdown, you can serve it directly without a conversion step.

1
import { notFound } from 'next/navigation';
2
import { getMarkdownContent } from '@/lib/content';
3

4
export async function GET(
5
  _req: Request,
6
  { params }: { params: Promise<{ slug?: string[] }> }
7
) {
8
  const { slug } = await params;
9
  const content = getMarkdownContent(slug?.join('/') ?? 'index');
10

11
  if (!content) {
12
    notFound();
13
  }
14

15
  return new Response(content, {
16
    headers: {
17
      'Content-Type': 'text/markdown',
18
    },
19
  });
20
}

The rich-text-to-markdown conversion preserves the content's structure. Code blocks keep their syntax highlighting markers, headings maintain their hierarchy, and links remain functional. The agent receives the same information as the HTML version, just in a format optimized for token efficiency.

Link to headingPerformance benefits

The HTML version of this page is around 500KB. The markdown version is 3KB, a 99.37% reduction in payload size. For agents operating under token limits, smaller payloads mean they can consume more content per request and spend their budget on actual information instead of markup.

We keep the HTML and markdown versions synchronized using Next.js 16 remote cache and shared slugs, so when content updates in our CMS, both versions refresh simultaneously.

Link to headingMarkdown sitemaps for agent discovery

Content negotiation also works for sitemaps. XML sitemaps are flat lists of URLs with no titles, no hierarchy, and no indication of what each page is about. A markdown sitemap gives agents a structured table of contents with human-readable titles and parent-child relationships, so they can understand what content exists on your site and navigate to what they need.

We serve markdown sitemaps for both our blog and documentation.

Here's the route handler we use to generate a markdown sitemap for blog posts:

1
import { getAllBlogPosts } from '@/app/content';
2

3
export const dynamic = 'force-static';
4

5
export async function GET() {
6
  const posts = await getAllBlogPosts();
7

8
  const lines = posts
9
    .sort((a, b) => new Date(b.date).getTime() - new Date(a.date).getTime())
10
    .map((post) => `- [${post.title}](/blog/${post.slug}.md)`);
11

12
  const sitemap = `# Blog sitemap\n\n${lines.join('\n')}`;
13

14
  return new Response(sitemap, {
15
    headers: { 'Content-Type': 'text/markdown' },
16
  });
17
}

For documentation or other content with nested sections, a recursive renderer preserves the hierarchy so agents understand which pages are children of which topics:

1
import { createTableOfContents, TocItem } from "@lib/content";
2

3
function renderTocItems(items: TocItem[], indent = '') {
4
  let sitemap = '';
5

6
  for (const item of items) {
7
    sitemap += `${indent}- [${item.title}](/${item.path})\n`;
8
    if (item.children) {
9
      sitemap += renderTocItems(item.children, `${indent}  `);
10
    }
11
  }
12

13
  return sitemap;
14
}
15

16
export async function GET() {
17
  const tableOfContents = createTableOfContents(`content`);
18

19
  const sitemap = `# Documentation sitemap\n\n${renderTocItems(tableOfContents)}`
20

21
  return new Response(sitemap, { headers: { 'Content-Type': 'text/markdown' } });
22
}

You can see this in action with the Vercel documentation sitemap.

For agents that don't send the Accept header, a link tag in your HTML <head> provides an alternative discovery path:

<link rel="alternate" type="text/markdown" title="LLM-friendly version" href="/llms.txt" />

Link to headingMaking your site agent-friendly

Content negotiation, markdown sitemaps, and link rel="alternate" tags give agents three ways to find and consume your content efficiently. You can read this page as markdown by sending Accept: text/markdown to any blog or changelog URL on vercel.com, as shown in the curl example above.

For an implementation reference, see how to serve documentation for agents in our knowledge base.