llms.txt on Webflow: What It Does and What It Doesn't

llms.txt is a plain-text Markdown file at the root of a website that gives AI agents a short, curated map of what the site contains and where the clean text lives. It was proposed by Jeremy Howard in September 2024 and reached version 2 of its specification on 10 August 2026. Think of it as a sitemap written for language models rather than for search crawlers — Webflow's own release note uses exactly that comparison.

And yes, Webflow supports it natively. You upload the file in your site's SEO settings; it has worked that way since 23 July 2025. We are writing this because a good deal of the Webflow-specific advice on llms.txt predates that release and still insists you need a reverse proxy to serve one at all. You do not. What you may still need a proxy for is everything around the file — and that distinction is the whole of this article.

We run all of it in production on this site, so the checks below are ones we ran on ourselves this morning, including the two we failed.

What llms.txt actually is

The specification lives at llmstxt.org. It is authored by Jeremy Howard, was first published on 3 September 2024, and its current revision — "The /llms.txt file, v2" — is dated 10 August 2026. The reasoning it opens with is a cost argument rather than a marketing one: an HTML page "wraps its information in navigation, ads, and JavaScript", converting it back to clean text is "difficult and imprecise", and "every wasted token costs time and money".

The format is deliberately boring. An H1 with the site or project name, an optional blockquote summary, then H2 sections of annotated links. One section may be named Optional, which carries a specific meaning: those links can be skipped when the agent needs a shorter context. It is Markdown rather than XML or JSON precisely because the intended reader is a language model.

Two things in v2 matter more than they look, and most write-ups predate them. First, the file no longer belongs only at the root: it "can be placed at the site root, or at any path within it, covering the pages under that path", so /insights/llms.txt is a legitimate, spec-sanctioned second file rather than a hack. Second, v2 recommends two link relations so clients can discover these files from the page itself: rel="alternate" type="text/markdown" pointing at a page's Markdown version, and rel="describedby" pointing at the llms.txt that covers it.

The spec also notes who eats their own cooking here: OpenAI, Anthropic and Gemini all publish llms.txt files for their own developer documentation.

Yes, Webflow serves llms.txt natively

This is the part that is most often reported wrongly, so here is the release note verbatim. On 23 July 2025, under the heading "Upload an llms.txt file for your site", Webflow wrote:

"You can now upload a llms.txt file for your site to help improve LLMs' understanding of your content. Webflow will make the llms.txt file you upload available in your domain's root, while also ensuring it does not get indexed by search engines."

And, on what the file is for: "Think of llms.txt like a sitemap — not a robots.txt file — but specifically designed for LLMs."

So: an upload field in your SEO settings, and Webflow handles both serving the file at the root and keeping it out of search results. That second part is not cosmetic, and we come back to it below because it is a check we fail.

There is an automation path as well, though a narrow one. Webflow's Beta Data API carries three endpoints for this file — GET, PATCH and DELETE on /beta/sites/:site_id/llms_txt, reading with the site_config:read scope — grouped under "Well-known files" beside robots.txt and 301 redirects, in the Enterprise section of the reference. The public release note names no plan requirement for the manual upload, so check your own site settings rather than taking a number from anyone's blog, ours included.

One genuine Webflow limitation survives all of this, and it is why proxies entered the conversation at all: Webflow serves pages, not files. Request a path Webflow does not know and you get its own 404 page as text/html — confirmed on our domain this morning, where a made-up .txt path returns a 906-byte HTML error page. The native llms.txt feature carves out one exception to that rule, not a general door.

What the native upload does not cover

Everything the specification asks for beyond "one file at the root" is still yours to solve. The native field takes a static file: it does not read your CMS, so it does not know you published an article this morning, and nothing re-writes it when you do. On a site that ships weekly, a hand-maintained index is wrong most of the time.

On this site those four jobs are done by a Cloudflare Worker sitting in front of Webflow, and you can tell which layer answered a request from the response headers alone — a useful diagnostic on any Webflow site behind a proxy. Our HTML pages come back carrying x-wf-region and x-lambda-id, Webflow's own hosting fingerprint. Our /llms.txt carries neither, and instead a plain cache-control: public, max-age=300. Different layer, same domain.

Read the table as a build-versus-buy sheet, not a verdict on Webflow. If all you want is a curated overview file, the native upload is the right answer — stop there.

Does anything actually read it?

Here is where we part company with most of the pages selling this. The honest answer is that nobody is obliged to read your llms.txt, and the largest search company on earth is on record saying it does not.

Asked on Reddit whether Google's guidance on the file was contradictory, Google's John Mueller replied that it is "purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?)". He also made a point that is hard to argue with: "if you use an LLM to create the file for you, doesn't that mean the LLM could just … create it for itself too?" His preferred direction is WebMCP, a Google-backed proposal for letting agents use a site's actual functionality rather than re-reading its text. Both quotes come from Search Engine Journal's write-up of the exchange, published 2 June 2026.

The Chrome Lighthouse audit people cite as Google endorsing the file is milder than its reputation too. As quoted in that same piece, it calls llms.txt an "emerging convention" and says that without one, agents "may spend more time crawling the site". A hedge, not a ranking factor.

Our own evidence points the same way, and it is evidence against us. We track 18 buying-intent queries in our niche weekly and capture the AI Overview on each. Across eight captures and 129 AI answers, this site earned seven citations — every one of them on our own company name. Zero on any commercial query. All of it was already live during that window: valid schema on every article, a curated llms.txt, a subpath index, Markdown twins, and open access for every AI crawler. The full dataset is in our eight-week AI Overview write-up.

So we cannot tell you llms.txt earns citations, because on our own site it demonstrably has not. What we can tell you is what Mueller put last and we would put first: "the most basic agentic optimization is in place, namely: don't block agents. I think that hurdle will be the biggest, for most sites."

The piece with a measurable payoff: Markdown twins

Strip out the speculation and one part of this has an effect you can put a number on. The specification asks for "a clean markdown version of those pages at the same URL as the original page", with .md appended or substituted. Unlike the citation question, the benefit here is arithmetic: a Markdown twin is the same article with the navigation, the CSS links, the analytics snippets and the interaction scripts removed.

We measured all fifteen articles on this blog this morning, fetching the HTML page and its .md twin and comparing transferred bytes. Across the set, 1,272,155 bytes of HTML became 187,338 bytes of Markdown — 14.7% of the original, an 85% reduction. Per article the ratio ran from 5.6x to 12.8x. The table below shows the two extremes and three articles in between, so you can see the spread rather than only our best case.

Bytes are not tokens, and we are not going to pretend they are. But the content is identical, so the direction is not in question, and it is the same argument Cloudflare makes for the feature it shipped natively: its Markdown for Agents documentation, last updated 13 July 2026, describes agents sending Accept: text/markdown and Cloudflare converting the page on the fly for enabled zones, returning token counts before and after in the response.

Content negotiation is worth testing whatever you serve. Ours honours it: request an article with Accept: text/markdown and you get the Markdown twin, not the page. That is our Worker rather than Cloudflare's feature — the give-away is that the response carries none of the token-count headers Cloudflare documents.

Six checks you can run on your own site today

None of these needs a tool or a subscription. We ran all six on this site while writing, and we publish our own results including the two failures.

1. Can AI crawlers reach you at all? Request one of your own pages with each agent's user-agent string and confirm a 200 rather than a 403 or a challenge page. Test the agents rather than reading your robots file — CDN bot-management rules block crawlers without ever appearing in robots.txt. Ours: GPTBot, ClaudeBot, PerplexityBot, Googlebot and Google-Extended all returned 200 on the HTML page, the .md twin and llms.txt. Pass.
2. Does your llms.txt come back as text? Check the content-type, not just the status code. A 200 that says text/html means you are looking at a page pretending to be a file. Ours: text/plain. Pass.
3. Does the file match the live site? Read your llms.txt beside the page it describes — pricing against the pricing page, tier names character by character. A stale index is worse than none, because it feeds wrong numbers to something that will repeat them confidently. Ours: the seven build packages and three care plans in llms.txt match our pricing page exactly. Pass.
4. Do your pages declare their Markdown version? Look for rel="alternate" type="text/markdown" in the head. Ours: present on every article. Pass.
5. Do your pages point at the llms.txt that covers them? This is the second relation v2 recommends, rel="describedby". Ours: missing. Fail — the spec revision is five weeks old and we have not shipped it yet. It is on our list.
6. Are your text files kept out of the index? Check for an X-Robots-Tag on llms.txt. Ours: absent. Fail, and here the native Webflow upload is simply better than our custom setup, because it noindexes the file for you.

Four passes and two failures, on a file family we built ourselves. That is the argument for running the checks rather than trusting the setup.

FAQ