llms.txt: Discovery for AI Documentation
- llms.txt is a machine-readable discovery format that organizes documentation pages and embeds agent instructions for efficient AI retrieval.
- It functions as a site-level index within a layered documentation stack, complementing files like AGENTS.md and skill.md without replacing them.
- Empirical studies show that llms.txt reduces fetch requests and assists navigation under token limits, enhancing overall documentation access.
LLMs.txt is an LLM-oriented, machine-readable discovery and instruction format that recent work places at the interface between documentation, retrieval, and agent behavior. In developer-portal settings, it is described as a directory-like index of documentation pages, with descriptions, intended to help AI agents discover where relevant information lives; in linked-data retrieval, “LLMs.txt-style agent instructions” appear as explicit guidance embedded in entity pages for LLM agents; and in broader interface design discussions, it is treated as part of a shift toward more structured, semantically explicit LLM interaction layers rather than undifferentiated plain text (Borysenko, 2 Apr 2026, Volpini et al., 11 Mar 2026, Hind et al., 23 Mar 2026).
1. Definition and conceptual scope
The literature does not treat LLMs.txt as a single, fully unified artifact across all settings. In documentation-portal research, it is framed as the discovery layer in a broader machine-readable stack: a site-level index that lists documentation pages and descriptions so that AI systems can find relevant material efficiently (Borysenko, 2 Apr 2026). In linked-data retrieval research, by contrast, LLMs.txt appears as a design pattern for explicit agent instructions embedded in web entity pages, rather than as a standalone retrieval system or a formal protocol implementation (Volpini et al., 11 Mar 2026).
This distinction is important. One line of work emphasizes LLMs.txt as a way to expose documentation entry points, stable links, and descriptions to AI coding agents and assistant services (Borysenko, 2 Apr 2026). Another uses “LLMs.txt-style agent instructions” as one component of an enhanced entity page that also contains Schema.org-typed HTML, embedded JSON-LD, visible linked entity navigation, content negotiation guidance, neural search references, and breadcrumbs (Volpini et al., 11 Mar 2026). A common element across both is that LLMs.txt is meant to reduce ambiguity about where relevant information is located and how an agent should use it.
A recurrent misconception is that LLMs.txt is being presented as a complete retrieval architecture. The linked-data study is explicit that it is not: the experimental evidence concerns a combined enhanced format, not LLMs.txt alone, and the paper does not isolate LLMs.txt as a standalone ablation (Volpini et al., 11 Mar 2026). The documentation study is equally explicit that LLMs.txt does not replace other files such as AGENTS.md, skill.md, or agent-permissions.json; it serves a narrower discovery function (Borysenko, 2 Apr 2026).
2. Place in the machine-readable documentation stack
One paper situates LLMs.txt within a layered governance and discovery stack for AI-driven documentation consumption. In that framing, different files support different aspects of agent interaction, and LLMs.txt is specifically the site-level discovery component (Borysenko, 2 Apr 2026).
| Artifact | Role in the stack |
|---|---|
robots.txt |
Classic crawler policy |
LLMs.txt |
Directory-like index for site-level discovery |
skill.md |
Structured capability description |
agent-permissions.json |
Declarative interaction governance |
AGENTS.md |
Repo-local instructions for coding agents |
| MCP-based feedback channels | In-environment issue reporting and feedback |
This decomposition clarifies what LLMs.txt is intended to do and what it is not intended to do. AGENTS.md is described as best for project-level instructions inside a repository, where an agent searches for it in the root and uses it as ongoing guidance. skill.md is described as a structured summary of what a product or API can do, what inputs it needs, and relevant constraints. agent-permissions.json is a declarative policy manifest for automated interactions, including rate limits, human-in-the-loop requirements, and preferred endpoints. By contrast, LLMs.txt is best for site-level discovery: it helps agents find the right documentation pages efficiently (Borysenko, 2 Apr 2026).
This layered view also places LLMs.txt in a governance context rather than a purely formatting context. The purpose is not merely to make pages easier to parse, but to create a predictable route by which agents can discover current documentation, direct retrieval toward stable entry points, and reduce wasted fetches under limited token budgets (Borysenko, 2 Apr 2026). This suggests that LLMs.txt is most useful when paired with canonical links, explicit capability summaries, policy manifests, and feedback channels rather than deployed in isolation.
3. Empirical motivations from agent access patterns
The need for LLMs.txt-style discovery is motivated empirically by how AI agents and assistant services actually consume documentation. A documentation-portal study reports that AI coding agents and assistant services often compress documentation consumption into one or two HTTP requests. In some cases maintainers may observe only one request, or a robots.txt request plus one content request, even when the content was fully consumed by the agent (Borysenko, 2 Apr 2026).
This behavior breaks assumptions inherited from human web navigation. Traditional measures such as session depth, time on page, click path, and bounce rate become unreliable indicators of actual documentation consumption when the agent performs server-side or single-fetch retrieval (Borysenko, 2 Apr 2026). The same study notes that common client-side analytics events—including click, scroll-tag, page_view, tutorial_begin, tutorial_complete, view_search_results, login, search, and select_content—disappear when content is fetched directly on the server side rather than traversed in a browser session (Borysenko, 2 Apr 2026).
The study further reports distinct HTTP behavioral signatures across nine AI coding agents and six AI assistant services, including differences in runtime environment, pre-fetch behavior, User-Agent, the presence or absence of Accept, the presence or absence of Sec-Fetch-*, and whether the system probes robots.txt before fetching content (Borysenko, 2 Apr 2026). This matters for LLMs.txt because agents that do not navigate menus or multi-page site hierarchies benefit from a concise, curated discovery index.
Token limits provide a second empirical motivation. The same paper presents the Cisco Secure Firewall Management Center REST API Quick Start Guide, Version 10.0 as an example of a documentation artifact with 193,217 tokens and 717,993 characters, using it to show that some documentation pages can exceed or strain the context window of many AI coding agents (Borysenko, 2 Apr 2026). In that setting, machine-readable discovery is valuable not only because it directs the agent toward relevant pages, but also because it supports decisions about whether to fetch a subsection, use a compressed summary, or start a fresh chat context.
These observations underwrite the paper’s argument that documentation should be designed for retrieval, not just for visual navigation. LLMs.txt is therefore justified less as a stylistic convention than as a response to single-fetch access patterns, limited context windows, and the collapse of conventional analytics visibility (Borysenko, 2 Apr 2026).
4. LLMs.txt-style instructions in retrieval and linked-data systems
In the linked-data retrieval literature, LLMs.txt appears not as a separate file but as a visible component of an enhanced entity page. The paper defines “LLMs.txt-style agent instructions” as a block of natural-language guidance included in the entity page to tell LLM agents how to use the page and its links. The appendix renders this as a dedicated <section class="agent-instructions"> containing <pre>{ llms_instructions }</pre> (Volpini et al., 11 Mar 2026).
The enhanced entity page is built on top of a Linked Data Platform and includes the following features: Schema.org-typed HTML, embedded JSON-LD, visible linked entity navigation, content negotiation guidance, LLMs.txt-style agent instructions, neural search SKILL reference for cross-entity discovery, and Schema.org type breadcrumbs for hierarchical context (Volpini et al., 11 Mar 2026). The paper’s conceptual claim is that the Linked Data Platform acts as an external memory layer for the agent, while the LLMs.txt-like instructions help the agent use that memory effectively.
The experimental design is a 3 × 2 factorial design, plus an Enhanced+ variant, across four domains, 158 entities, 349 test queries, and 2,439 valid results after excluding four error cases. It compares plain HTML, HTML with JSON-LD, and an enhanced entity page under both standard RAG and agentic RAG, with the agentic configuration using a Google ADK agent in a ReAct-style loop and the tools search_documents, follow_entity_link, and search_knowledge_graph (Volpini et al., 11 Mar 2026).
The reported results are nuanced. JSON-LD alone yields only modest improvements: under standard RAG, C2 3.89 vs C1 3.62 in accuracy with p_adj = 0.024 and d = 0.18, while completeness improvement is not significant after correction (Volpini et al., 11 Mar 2026). By contrast, the enhanced entity page produces the largest gains. Under standard RAG, C3 4.69 vs C1 3.62 in accuracy and C3 4.45 vs C1 3.01 in completeness, reported as about +29.6% accuracy improvement. Under the full agentic pipeline, the abstract reports +29.8% accuracy improvement. The Enhanced+ variant achieves the highest absolute scores, with accuracy: 4.85/5 and completeness: 4.55/5, but the incremental gain over the base enhanced format is not statistically significant (Volpini et al., 11 Mar 2026).
For the interpretation of LLMs.txt, the key point is that the paper does not claim that LLMs.txt-style instructions alone are the main source of improvement. Instead, it states that they are part of a broader enhanced design that materializes linked data, exposes relationships, and makes navigation visible. The evidence therefore supports the usefulness of LLMs.txt-style instructions as a supportive component of agent-readable pages, but not as an independently decisive factor (Volpini et al., 11 Mar 2026).
5. Relation to LLM-native interface languages
A separate line of work on LLM interfaces places LLMs.txt in a wider movement toward explicit structure and semantics at the LLM boundary. The paper introducing LLMON argues that plain-text prompts are weak because they collapse instructions, data, examples, and constraints into a single flat string, leading to confusion, weak referential structure, poor portability, non-determinism, and prompt-injection risks (Hind et al., 23 Mar 2026).
LLMON is presented as an LLM-native markup language designed to make structure, semantics, and execution intent explicit through user-defined tags, named instances, explicit nesting names, prefix annotation, JSON convertibility, and a machine-facing form that avoids escaping by using special tokens (Hind et al., 23 Mar 2026). It supports distinctions such as instr, data, and exec, along with references like exec:y.instr and exec:y.input, so that embedded malicious natural-language instructions inside a data span should be treated as data rather than executable control (Hind et al., 23 Mar 2026).
The paper explicitly relates this agenda to an LLMs.txt-style format. It states that LLMON is “very much in the spirit of an LLMs.txt-style idea,” but goes further because it is LLM-native, encodes semantics, not just formatting, supports references, nesting, and execution intent, is meant to be consumed directly by models and runtimes, and is convertible from common structured formats like JSON (Hind et al., 23 Mar 2026). It also reports preliminary gains from LLMON-structured training and inference-time masking on a custom Distractor benchmark, including 74.2 percentage points average improvement across training experiments and 29.3 percentage points average gain for inference-time masking (Hind et al., 23 Mar 2026).
This suggests a useful conceptual distinction. LLMs.txt addresses document and site discovery, while LLMON addresses typed control at the model interface. Both emerge from dissatisfaction with flat text, but they operate at different levels. LLMs.txt tells an agent where relevant materials are and may provide guidance on their use; LLMON specifies, within the prompt or runtime artifact itself, what is instruction, what is data, and what execution binding should govern generation (Hind et al., 23 Mar 2026).
6. Limitations, misconceptions, and open directions
Current evidence supports a restrained interpretation of LLMs.txt. The linked-data retrieval study does not isolate LLMs.txt as a standalone ablation and therefore does not show that it independently drives a statistically meaningful improvement beyond the broader enhanced linked-data presentation (Volpini et al., 11 Mar 2026). The documentation study likewise does not present a standalone benchmark for LLMs.txt; instead, it implies evaluation criteria such as the number of requests per documentation task, time to first relevant fetch, token cost of retrieval, reduction in irrelevant page fetches, and whether agents find the intended documentation in fewer requests (Borysenko, 2 Apr 2026).
Another limitation is that machine-readable discovery is not enough by itself. The documentation-portal paper argues that teams should pair LLMs.txt with stable canonical links, token counts or size indicators, AGENTS.md in repositories, skill.md for capability summaries, agent-permissions.json where policy matters, and MCP-based feedback channels for in-environment issue reporting (Borysenko, 2 Apr 2026). This reflects the paper’s ecosystem view that discovery, capability description, governance, and feedback are separate but complementary functions.
A further misconception is to treat LLMs.txt as a substitute for structure deeper in the LLM pipeline. The LLMON paper argues that reliable, safe, and reusable LLM use requires explicit structure and semantic metadata at the interface itself, not merely better-formatted source material (Hind et al., 23 Mar 2026). From that perspective, LLMs.txt improves discoverability and, in some settings, provides agent-facing instructions, but it does not by itself solve instruction–data separation, execution control, or prompt-injection defense.
The open direction common to these papers is a move away from opaque flat text toward explicit, machine-usable structure. In one branch this takes the form of AI-native documentation stacks with LLMs.txt as the discovery layer; in another it takes the form of enhanced entity pages with LLMs.txt-style instructions embedded alongside linked data; and in a third it motivates LLM-native markup languages such as LLMON that introduce type-like semantics at training, prompting, and inference time (Borysenko, 2 Apr 2026, Volpini et al., 11 Mar 2026, Hind et al., 23 Mar 2026).