What Is llms.txt? An Operator's Definition

What is llms.txt?

llms.txt is a proposed plain-text file, published at a website’s root (/llms.txt), written in Markdown, that lists a site’s key pages with a short description of each. The idea, put forward in 2024, is that a large language model retrieving information about a site can read one short file instead of crawling the whole thing.

That is the dictionary definition, and it is where most explanations of llms txt stop. It is also, on its own, close to useless to an operator, because the file’s value depends entirely on who reads it — and that list is much shorter than the pitch decks selling “llms.txt optimisation” imply.

The proposal itself is simple by design: a Markdown file, a short introduction, and a set of links grouped under headings, so a system reading it gets a map of the site rather than a raw crawl. Nothing about the format is proprietary or hard to build; writing one by hand is a formatting exercise, not an engineering one.

What does publishing llms.txt actually change?

For most sites, publishing an llms.txt file changes almost nothing measurable, because only one major AI system has confirmed it reads the file at all. Perplexity has stated support for llms.txt. Google, OpenAI and Anthropic have not confirmed that Gemini, ChatGPT or Claude read or act on it.

That is the operator-level consequence, and it is different from the dictionary meaning. A Tuesday spent writing an llms.txt file is a Tuesday spent building an index for one confirmed reader, not the general-purpose AI-visibility lever it is often sold as. This matters because Pointerflow’s own service page for AI search visibility is explicit that we serve one at /llms.txt on our own site — because it is cheap, not because it moves rankings. That distinction is the whole article.

The actual lever for AI search visibility sits somewhere else entirely. AI-driven traffic to Shopify stores grew 8× year over year, and orders originating from AI-powered search grew nearly 13× (vendor-reported, Shopify president Harley Finkelstein, Q1 2026 earnings call). Separately, GEO (Generative Engine Optimization) techniques — answer-first structure, named sources, entity clarity — have been measured to lift a page’s visibility in AI-generated responses by 30–40% in a Princeton University study. Neither of those figures has anything to do with llms.txt. Both are about what is on the page itself.

What happens, measurably, before and after you publish llms.txt?

No vendor or independent source has published a before-and-after measurement of what publishing an llms.txt file actually does to AI citations or referral traffic — every number attached to “llms.txt results” in circulation is asserted, not measured, which makes the honest figure here — metric to confirm — not a made-up range. What follows is not a substitute for that measurement. It is the method for running it yourself, because nobody outside your own analytics stack can produce it for you.

The scope has to match the one confirmed reader. Perplexity’s crawler identifies itself in server logs as PerplexityBot, and it is the only automated system with stated support for the file, so a test that mixes in general AI search traffic — ChatGPT, Gemini or Copilot referrals that have nothing to do with llms.txt — will not isolate anything. Filter to sessions where the referrer domain is perplexity.ai before doing anything else.

Confirm the fetch before trusting anything downstream of it. Search the server access logs for requests to /llms.txt from a user agent containing PerplexityBot in the days after publication. If that request never appears, the rest of the test is measuring noise, because nothing has actually read the file yet.

With the fetch confirmed, take a baseline: perplexity.ai referral sessions over a fixed window before publishing — 4 to 6 weeks is usually enough to smooth week-to-week variance without the window running so long that unrelated page changes contaminate it. Hold the pages listed in the file otherwise unchanged, publish, wait the same window length, and compare the same referral segment against the baseline.

A traffic delta alone will still miss the more direct signal: whether the file changed what gets cited. Run the actual queries your llms.txt-listed pages are meant to answer, directly in Perplexity, and record whether a citation appears, which URL it links to, and whether that URL is one you listed. A citation without a click leaves no trace in referral data at all, so this step catches what the traffic number cannot.

None of this produces a figure Pointerflow, or anyone else, can hand you in advance — the result depends on your catalogue, your existing page structure and the queries your customers actually type. But it is the only honest way to know whether the file did anything for your site, and it costs an afternoon, not a study.

Where do people get llms.txt wrong?

The most common mistake is treating llms.txt as a submission mechanism, the way sitemap.xml is submitted to Google Search Console to request indexing. It is not that. There is no submission step, no confirmation, and — outside Perplexity’s stated support — no verified system reading the file. An llms.txt file sitting at your root does not notify anyone that it exists.

The second mistake is sequencing: publishing llms.txt before the pages it points to are actually extractable. A model that does load the file and follows a link to a page still needs that page to answer a question in its first sentence, name real entities like Shopify or Klaviyo instead of “the platform”, and carry a sourced figure rather than an unattributed one. An llms.txt file pointing at pages that do not do this has curated a list of nothing worth citing.

How do you generate and maintain llms.txt for a Shopify catalogue with thousands of SKUs?

You do not list the SKUs. llms.txt is specified as a short, curated map of a site’s most useful pages, not a comprehensive index — that job already belongs to sitemap.xml, which every Shopify store regenerates automatically as the catalogue changes. A store with thousands of products should point llms.txt at collection pages, a small set of flagship or best-selling products, and content pages such as buying guides, not at every product URL the store has ever published.

Building that shortlist by hand works for an initial pass, but it does not survive routine catalogue turnover — nobody remembers to update a hand-curated file every time a product is discontinued or a collection is renamed, so it drifts out of date and eventually gets abandoned. The practical fix is to generate it from a signal that already exists in most Shopify admin workflows: a metafield or tag — “featured”, “bestseller”, whatever a merchandising team already uses to flag priority products — queried through the Admin API and written into /llms.txt by a scheduled job rather than typed by a person. Shopify’s theme layer has no native llms.txt generator, so this has to live outside the theme: a Shopify Flow automation, a small serverless function, or a build step if the storefront runs headless.

Stability matters more than completeness at the URL level. A hard-coded product link breaks the moment that SKU is discontinued, renamed, or merged into a duplicate listing — all routine events in a catalogue of thousands. A collection URL survives all three, because the collection persists even as the products inside it turn over. Where the choice exists, list the collection and let it carry the traffic to whatever is currently in it, rather than a product page that will eventually 404.

The single largest maintenance risk is silence. Sitemap.xml has an entire tooling ecosystem — Search Console, crawl reports — built around flagging when something in it breaks. llms.txt has none of that. A stale file sits at the root, technically present, quietly pointing at pages that no longer exist or no longer matter, and nothing tells anyone. The fix is not a person remembering to check it; it is wiring the regeneration into whatever process already regenerates the sitemap on catalogue changes, so llms.txt inherits that trigger for free instead of depending on a calendar reminder nobody honours after the second quarter.

How is llms.txt different from robots.txt and sitemap.xml?

llms.txt is neither an access-control file nor a full page index, which is what makes it easy to confuse with the two standards it sits between. robots.txt is a permission file — it tells crawlers which parts of a site they may or may not access, and every major search and AI crawler respects it, because ignoring it is a compliance failure, not a missed opportunity. sitemap.xml is a machine-readable, comprehensive list of a site’s URLs, submitted to search engines to aid indexing, and Google actively consumes it.

llms.txt does neither job. It is a short, hand-curated, human-readable Markdown summary of a site’s most useful pages, with no enforcement and — again — one confirmed reader. Calling it “the robots.txt for AI” gets the mechanism backwards: robots.txt works because crawlers are contractually obliged to check it. llms.txt works only for whichever systems have chosen to read it, and today that is a much shorter list than the phrase implies.

How do you validate an llms.txt file, and what breaks it?

Validate it by testing every listed URL directly, because nothing else will tell you it is broken. There is no submission step and no confirmation from Perplexity, or anyone else, that a file has been read, so the only way to know a link in it still works is to check the link, not to trust that publishing it was the end of the job. Three failure modes account for nearly everything that goes wrong, and none of them throws an error you would ever see without looking.

A redirect breaks the link silently. A URL in llms.txt that returns a 301 or 302 instead of loading directly adds a hop that not every crawler follows, and even where it is followed, the crawler ends up reading the destination under a different URL than the one you told it to expect. List the destination URL directly rather than relying on the redirect to carry the crawler there.

An authentication wall is the more embarrassing failure to ship, because it usually comes from copying a link out of an internal document — a staging domain, a password-protected preview, a gated resource behind a login. Any of those resolves to a login screen rather than content for a crawler with no session, which contributes nothing readable and sends a false signal that the page exists when, for practical purposes, it does not. Every URL in llms.txt has to be a fully public, production URL with no wall in front of it.

A stale link is the failure mode specific to a catalogue that changes: a Shopify product handle renamed, a duplicate SKU merged into another listing, an item discontinued outright, all turn a previously valid entry into a 404. Unlike an orphaned page flagged in Search Console, nothing surfaces a broken llms.txt entry to you — it sits there until a person happens to click it.

A fourth condition is worth checking even though it is not strictly a break in the file itself: a URL listed in llms.txt that is also disallowed in robots.txt sends two files contradictory instructions to any crawler that respects both. Every major crawler respects robots.txt, which is precisely why it works as an access-control mechanism at all — so a crawler told not to fetch a page will not fetch it, no matter how prominently llms.txt points there.

Because none of this validates itself, script it. A scheduled job — a CI action, a Shopify Flow trigger, or a short script run before each deploy — that sends a request to every URL in the file and flags anything that is not a clean 200 catches a redirect, a 404 or a robots.txt conflict before a person would ever notice one by hand. Tie that job to whatever step regenerates the file in the first place, so the two run together and a broken link never ships silently.

None of this makes llms.txt worthless. It costs little to publish and there is no real downside, provided nobody mistakes it for the fix. The actual problem it gets bundled into is broader: whether a brand’s content is structured so that any AI system — Perplexity today, potentially others later — can find a correct, attributable passage to cite. That is an AI search visibility problem, not a file-format problem, and it is the reason AI search visibility work starts with page structure and named sources rather than with a root-level text file.