What Is an llms.txt File? (And What It Does Not Do)

Q: What is an llms.txt file?

A markdown summary placed at `/llms.txt` so software that already fetched your site can read a short entity description and a list of docs URLs. It is not robots.txt, not a sitemap, and not a ticket into Google AI Overviews or ChatGPT recommendations. KeywordTool’s question form is 90/month; this slug already exists for it.

An llms.txt file is a markdown summary at /llms.txt. It does not rank you in Google or ChatGPT. Extra AI files do not get you into AI Overviews or AI Mode. Ship robots.txt, a sitemap, and indexed HTML instead.

Updated 10 Sep 2026. Checked again 4 Oct 2026: the live file reported 36 tools, and the registry line still said “once live.”

What we tested (10 Sep 2026)

We used Scope's own files as the worked example — a SaaS that already ships /llms.txt and /llms-full.txt, not a hypothetical dentist.

Google Search Console 28d (site): 38 clicks, 7,068 impressions, CTR 0.54%, average position 42.33.
This URL: 10 impressions, 0 clicks, average position 7.2. No query row matching llms survived anonymization.
URL Inspection: Submitted and indexed. Last crawl 30 Jul 2026 07:18 UTC — 42 days before this rewrite. User canonical = Google canonical = https://scope.online/learn/llms-txt-complete-guide. Referring URL = the XML sitemap. Rich results detected: Breadcrumbs only.
Sibling definition page: /glossary/llms-txt is Crawled — currently not indexed (last crawl 20 Apr 2026). That is why this slug stays the essay. We are not minting /learn/what-is-llms-txt.
Live /llms.txt: HTTP 200, 6,184 bytes, text/plain. Inspection: Crawled — currently not indexed, last crawl 1 Aug 2026, referring URL = homepage. The file still says “27 tools” and “Official MCP Registry name (once live)”.
Live /llms-full.txt: HTTP 200, 1,365,150 bytes.
Live /robots.txt: HTTP 200, 135 bytes. User-Agent: * / Allow: / / sitemap line present. No named GPTBot, OAI-SearchBot, or ClaudeBot stanzas. The wildcard allows those crawlers; a later Disallow would be the failure.
KeywordTool (US, production, not sandbox, rolling bank 9 Sep 2026): “what is llms txt file” 90/mo; “llms txt for seo” 20/mo; “llms.txt vs robots.txt” 20/mo; “llms.txt vs llms-full.txt” 20/mo. The 6,600/mo head term llms.txt is a preposition dump — it does not pick this title.
Google SERP, desktop US, 10 Sep 2026: this URL not in the top 8–10 for any of what is llms txt file (llmstxt.org #1, Semrush #3, Ahrefs #8), llms.txt vs robots.txt (Reddit #1, Search Engine Land #2), llms txt for seo (Reddit #1, HubSpot community #2), or does llms.txt work (Reddit #1, SE Ranking “Why It Doesn't Work” #2).
Answer engines this run: ChatGPT.com HTTP 403, Perplexity.ai HTTP 403, grok.com and Gemini HTTP 200 HTML shells. No citation list was captured. Do not invent one.

That gap — Google indexed the April “complete guide,” last crawled it in July, and still sends 10 impressions while llmstxt.org owns the definition — is why this page exists.

What is an llms.txt file?

llms.txt is a plain-text markdown file at https://yourdomain.com/llms.txt. Jeremy Howard (fast.ai) proposed it in 2024. The public spec still lives at llmstxt.org, which Google ranked #1 for “what is llms txt file” on 10 Sep 2026.

The file is a summary for software that already fetched your site, not a permission layer and not a sitemap. A typical layout:

# Product name

> One-line description

Two or three sentences of entity copy: who you are, who you serve, what you sell.

## Docs
- Getting started: https://example.com/docs

That is the whole invention. There is no official validator. There is no IETF RFC. OpenAI, Anthropic, and Google have not published “we read /llms.txt to decide who to recommend.”

The convention also mentions llms-full.txt: a longer dump. Scope's is 1.36 MB this morning. Length is not evidence of retrieval.

Does llms.txt work for SEO or ChatGPT?

No as a ranking lever. Maybe as optional agent documentation.

Google Search Central's AI features page, confirmed 2 Sep 2026, is explicit: you do not need extra machine-readable files, AI text files, or special schema to appear in AI Overviews or AI Mode. The page must be indexed and snippet-eligible. That is Google SEO. Keep doing it.

Two public tests we cite, not restated as our crawler log:

Webappski (13 Jul 2026): one site, 90 days, 84 of 62,100 AI crawler hits requested llms.txt — about 0.1%.
Concurate (11 Aug 2026): adding the file did not move AI Overviews; ChatGPT/Claude results were mixed; ordinary SEO still dominated.

Those are named, dated, and small. They are enough to kill the “add this file and ChatGPT will recommend you” claim. They are not a 10,000-query study, and this page does not invent one.

Common misconception

The April version of this URL opened with: “llms.txt is one of the simplest, most overlooked opportunities in AI visibility optimization.” That sentence is the counter-example. It is the tendon-rest advice of this topic. We left the slug in place and rewrote the claim.

Search Engine Land still frames the file as a “treasure map for AI.” GitBook and Webflow University still teach it as AEO homework. Reddit threads for “does llms.txt work” and “llms txt for seo” outrank vendor how-tos — which is the actual SERP, not a reason to clone their homework.

If you already have an SEO program, do not pause it to generate this file. If you already shipped one, do not delete it in a panic. Just stop treating it as the ChatGPT trick.

How is llms.txt different from robots.txt?

robots.txt is the access file. llms.txt is a brochure.

	robots.txt	sitemap.xml	llms.txt
Job	Allow or deny crawlers	List URLs + lastmod	Optional markdown summary
Who uses it	Every crawler that respects robots	Google, Bing, most AI crawlers	Unconfirmed for ranking / answers
Failure mode	`Disallow: /` hides you	Stale or tiny sitemap hides new URLs	Stale copy misleads an agent that bothered to fetch it
Google AI Overviews / AI Mode	Must allow the fetch	Helps discovery	Not required (Search Central, 2 Sep 2026)

KeywordTool's “llms.txt vs robots.txt” is 20/month. The pointed answer: robots decides whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Googlebot may enter. llms.txt does not. OpenAI's own docs are blunt about OAI-SearchBot: block it and you are absent from ChatGPT search answers — not ranked low, absent. That is a robots.txt ticket, not an llms.txt ticket.

Scope's live robots.txt this morning is 135 bytes:

User-Agent: *
Allow: /
Disallow: /app/
Disallow: /b/
Disallow: /admin/
Disallow: /settings/

Sitemap: https://scope.online/sitemap.xml

No named AI-bot stanzas. The * allow is enough unless a later group blocks those user-agents. Named allows are documentation for humans. A Disallow is the failure.

What is llms-full.txt vs llms.txt?

llms.txt is supposed to be short. llms-full.txt is supposed to be the dump.

On 10 Sep 2026 Scope shipped both: 6,184 bytes vs 1,365,150 bytes. Google had crawled the small file (1 Aug) and still marked it not indexed. That is the correct outcome for a text/plain dump. It is not a Learn article, not a rich result, and not a substitute for the HTML page you want cited.

If you generate a “full” file from every MDX slug, you will ship a megabyte of yesterday's copy. Agents that want to operate a product should use a protocol, not parse a novel.

How is llms.txt different from MCP?

MCP is a live tool interface. llms.txt is a static README.

Scope MCP lets Claude, ChatGPT, or Cursor call get_visibility_report, get_citations, and run_scan against a business you own. The agent gets current scores, not a markdown brochure from whenever marketing last regenerated the file.

KeywordTool lists “llms txt vs mcp” as a 10/month preposition. The split:

Ship MCP (or an equivalent API) when an agent needs to do work.
Ship llms.txt only if you want a human-readable map of docs URLs — the Stripe / Vercel pattern.
Do not expect ChatGPT's consumer answers to open either file before they name a brand. Off-site consensus and extractable HTML still move citations (Ahrefs, Jul 2026). Claude still leans on Brave's top 10 (~79% in Profound's Jul 2026 split). Those are ranking and distribution facts. They are not an llms.txt feature.

This morning's Scope /llms.txt is the worked example of drift: the heading says 27 tools, the live MCP surface is not a 27-tool brochure, and the registry line still says “once live.” A stale map is worse than no map. We are not fixing that file in this article. The stale lines are the evidence.

What should you do instead?

Plumbing that actually gates retrieval, in order:

Allow the crawlers. robots.txt must not block GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Googlebot on the money URLs.
Give them a sitemap with lastmod. Live /sitemap.xml this morning had 333 <loc> values, matching GSC's last download. Discovery is the XML file, not the markdown dump.
Ship HTML Google can index. This URL is Submitted and indexed, last crawled 30 Jul 2026, 10 impressions. Indexed ≠ ranking. The sibling JSON-LD guide already showed six schema blocks in the HTML and Breadcrumbs-only on inspect. Extra files do not skip that queue.
Put the entity in the first 80 words of the page you want cited. Models extract; they do not hunt a side file first.
Measure recommendations separately from rankings. Search Console for Google. An AEO audit / AI visibility checker for whether ChatGPT, Claude, Gemini, or Perplexity name you. Complementary, not a replacement.

Off-site work still sits next to this: the lists, videos, and independent pages engines already cite. Do not cold-email those publishers from your marketing domain. Do not pause Google SEO to write a treasure map.

Worked example: Scope, 10 Sep 2026

Not a hypothetical local shop. This is the SaaS that publishes this page.

Check	Result 10 Sep 2026
Learn URL HTTP	200, self-canonical
GSC coverage	Submitted and indexed, last crawl 30 Jul 2026
GSC 28d	10 impressions, 0 clicks, pos 7.2
Definition SERP	Absent. llmstxt.org #1, Semrush #3, Ahrefs #8
`/llms.txt`	200, 6,184 B, crawled-not-indexed, “27 tools”, “once live”
`/llms-full.txt`	200, 1.36 MB
robots.txt	200, `Allow: /`, sitemap line, no named AI bots
ChatGPT / Perplexity web	403 this morning
Claim we will not make	“Our llms.txt is why ChatGPT will recommend Scope”

If shipping the file were the overlooked opportunity, this URL would not be sitting at 10 impressions under an April title while the spec site ranks #1. The honest use of our file is agent documentation, and even there it is out of date. The honest SEO/AEO work is indexation, extractable HTML, citations, and off-site consensus — the same stack as GEO vs SEO.

30-minute operator checklist

Do this on one money URL, not the whole site.

Run Search Console URL Inspection on the HTML page, not on /llms.txt. Write coverage, last crawl, Google canonical. Ours: indexed, crawl 30 Jul 2026, canonical self.
Search what is llms txt file and does llms.txt work. Write whether your URL is in the top 10. Ours was not, on 10 Sep 2026.
curl /robots.txt. Confirm you are not Disallowing GPTBot / OAI-SearchBot / ClaudeBot. Named Allow stanzas are optional if User-Agent: * already allows /.
curl /sitemap.xml. Count <loc>. Confirm the money URL is in the file. Ours: 333, including this slug.
Fetch the money URL with JavaScript off. Confirm the first 80 words state who you are and what you do.
If /llms.txt exists, open it. Check HTTP 200, text/plain, and whether the entity sentence and any tool counts still match the product. Ours failed the last check (“27 tools”, “once live”).
Do not spend the rest of the half-hour generating llms-full.txt. Spend it on one unique dated fact on the HTML page, then on one off-site page that already gets cited.
Measure the recommendation layer with the AI visibility checker or Scope MCP. Record cited / mentioned / omitted / blocked. Blocked is valid. ChatGPT and Perplexity were 403 this morning.

Key takeaways

An llms.txt file is a markdown summary at /llms.txt. It is not robots.txt, not a sitemap, and not a ChatGPT ranking trick.
Google Search Central (2 Sep 2026): no extra machine-readable files or AI text files to appear in AI Overviews or AI Mode. Indexed, snippet-eligible HTML is the floor. Keep doing Google SEO.
On 10 Sep 2026 this URL was Submitted and indexed (last crawl 30 Jul) with 10 impressions, 0 clicks, and missing from Google's top results for “what is llms txt file.” llmstxt.org ranked #1.
Scope already ships /llms.txt (6,184 bytes, still “27 tools” / “once live”) and /llms-full.txt (1.36 MB). The small file is crawled-not-indexed. Shipping the file did not make this guide rank.
Webappski (Jul 2026): ~0.1% of AI crawler hits touched llms.txt. Treat the file as optional agent docs, then go back to robots, sitemap, SSR, entity copy, and citations.
MCP is how agents operate Scope. llms.txt is how they read a brochure. Do not confuse them.

References

Google Search Central, AI features (confirmed 2 Sep 2026): no extra machine-readable files, AI text files, or special schema to appear in AI Overviews or AI Mode.
Jeremy Howard, llms.txt proposal (Sep 2024). Google #1 for “what is llms txt file” on 10 Sep 2026.
Google Search Console URL Inspection of this URL, 10 Sep 2026: submitted and indexed; last crawl 30 Jul 2026 07:18 UTC; canonical self; rich results = Breadcrumbs. /llms.txt crawled-not-indexed (crawl 1 Aug 2026). /glossary/llms-txt crawled-not-indexed (crawl 20 Apr 2026).
Markifact GSC 28d, 10 Sep 2026: site 38 clicks / 7,068 impressions; this URL 10 impressions / 0 clicks / pos 7.2.
Markifact Google SERP, desktop US, 10 Sep 2026: what is llms txt file — llmstxt.org #1, Semrush #3, Ahrefs #8, Scope absent. does llms.txt work — SE Ranking “Why It Doesn't Work” #2.
KeywordTool.io production questions + prepositions, US/English, rolling bank 9 Sep 2026: “what is llms txt file” 90/mo, “llms.txt vs robots.txt” 20/mo, “llms txt for seo” 20/mo.
Webappski, Nobody reads llms.txt — 0.1% of AI traffic (13 Jul 2026): 84 / 62,100 crawler hits.
Concurate, Does llms.txt Boost AI Visibility? We Tested It (11 Aug 2026).
Ahrefs, What Is llms.txt, and Should You Care About It? (on the 10 Sep SERP).
Profound, July 2026 platform citation split (Claude / Brave ~79% vs ChatGPT lists-YouTube-UGC).
Ahrefs, July 2026, off-site consensus vs on-page length for AI citations.
Scope citation panel, 10 Sep 2026: ChatGPT/Perplexity 403; Grok/Gemini 200 shells; branded compare #4.

Q: What is an llms.txt file? A: A markdown summary placed at /llms.txt so software that already fetched your site can read a short entity description and a list of docs URLs. It is not robots.txt, not a sitemap, and not a ticket into Google AI Overviews or ChatGPT recommendations. KeywordTool’s question form is 90/month; this slug already exists for it.

Q: Does Google read llms.txt? A: Googlebot can fetch any public text file. Search Central (2 Sep 2026) says extra machine-readable files and AI text files are not how you appear in AI Overviews or AI Mode. On 10 Sep 2026 Google had crawled Scope’s /llms.txt (1 Aug) and marked it crawled-not-indexed. That is expected for text/plain.

Q: Does llms.txt help SEO? A: Not as a ranking factor. “llms txt for seo” is 20/month of people asking this. Keep the HTML page indexed, snippet-eligible, and internally linked. Use Search Console for rankings. Use an AEO audit for whether answer engines name you.

Q: What is the difference between llms.txt and robots.txt? A: robots.txt allows or denies crawlers. llms.txt is an optional brochure. Block OAI-SearchBot and you vanish from ChatGPT search. Forget llms.txt and Google SEO does not notice. Scope’s robots.txt on 10 Sep 2026 is a 135-byte User-Agent: * / Allow: / file with a sitemap line.

Q: Should I add llms-full.txt? A: Only if you have a reason to dump the whole site as markdown for agents. Scope’s file is 1.36 MB this morning. Google does not need it. An MCP server or a docs index is a better interface than a megabyte of generated prose.

Q: Will llms.txt get ChatGPT to recommend my business? A: No. Named tests (Webappski Jul 2026, ~0.1% of crawler hits; Concurate Aug 2026, no AI Overviews lift) and Google’s own AI-features note all point the same way. Off-site consensus and extractable first 80 words still matter. ChatGPT.com was HTTP 403 this morning — we did not invent a citation list.

Q: Do I still need Google SEO if I add llms.txt? A: Yes. Answer engines retrieve pages. Technical SEO, content, and links still determine whether you are eligible to be cited. This page is additive to SEO, not a replacement. Keep Search Console; add the AI visibility checker for the recommendation layer.

Q: How is this different from Scope MCP? A: MCP is a live tool protocol so an agent can pull visibility, citations, and scans. llms.txt is a static map of URLs. Use Scope MCP to operate the product. Use llms.txt only as optional docs hygiene, and keep the copy in sync. On 10 Sep 2026 the live file said “27 tools” and “once live.” On 4 Oct 2026 it reported 36 tools, and the registry line still said “once live.”