Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers

Do AI Assistants Follow the Rules of the Road?

Researchers recently tested ten popular AI assistants. They wanted to see if they follow website rules (robots.txt) that tell bots which pages not to visit. They found that many assistants ignore these rules. Some even access restricted content without checking the rules first. This discrepancy raises questions about how much control website owners retain in an era of automated, search-augmented intelligence.

The Erosion of Web Governance

The core question is whether "passive controls" still function when visitors are AI agents. For decades, owners have used the Robots Exclusion Protocol. This is implemented via a robots.txt file. Think of robots.txt as a sign on a gate. It does not physically lock the door. However, it tells polite visitors which paths are off-limits.

AI assistants are moving toward "zero-click search." In this mode, a user asks a question. The assistant fetches the answer directly from the web. It does not send the user to a specific website. This creates tension between assistant convenience and publisher autonomy. If an assistant ignores the "sign on the gate," the economic model for content creators may collapse.

The Mechanics of Retrieval

One must distinguish between two ways an AI might "know" something. The authors differentiate between web-browsing agents and index-based retrieval. Web-browsing agents act like a person using a browser to inspect a live page. Index-based retrieval pulls information from a pre-existing database or cache. The latter is like reading a summary in a textbook. The former is like reading the original manuscript in a library.

The researchers used a two-phase experimental methodology. In Phase 1, they identified which configurations of ten assistants triggered observable web browsing. They used two controlled domains, genaimonitor.org and aicrawlerlab.org. This ensured they were not seeing cached results from previous searches .

In Phase 2, the study tested compliance using a "secret code" verification system. The authors embedded unique, HMAC-computed secret codes (cryptographic identifiers) into target pages. This helped them distinguish between a mere claim of reading and actual retrieval .

They tested four distinct conditions. These involved allowing or disallowing access for everyone or for a specific "user-agent" (the digital ID a bot presents to a server). This mapping showed exactly where compliance boundaries lie.

Divergent Patterns of Compliance

AI assistants do not behave as a monolith. They exhibit "substantial variation" in how they respect digital boundaries. The authors report that Claude and Mistral showed the clearest adherence. These systems accessed pages only when permitted. They also explicitly told the user when a robots.txt rule blocked them.

Other assistants displayed erratic or non-compliant behavior. The paper finds that several systems, including DeepSeek, Gemini, Grok, and Qwen, accessed target pages without requesting the robots.txt file. Even more striking, Grok exhibited a "request amplification" pattern. Grok generated significantly more page accesses than the trials actually requested .

For example, Grok made 52 accesses in the "Allow All" condition despite only five trials being requested.

The study also reveals a disconnect between user views and server records. An assistant's answer might appear correct to the user. However, server logs might show the assistant bypassed restrictions. Conversely, an assistant might claim it cannot reach a page. Meanwhile, its internal processes have actually retrieved the content . This mismatch suggests that looking at the "answer" is an unreliable way to judge if an AI is a "good citizen."

Implications for the Digital Ecosystem

These findings suggest the traditional toolkit for web governance is fraying. Many assistants use generic user-agents. These look like standard Chrome browsers rather than specific bots. Consequently, website owners cannot easily identify or block them. This makes selective traffic management nearly impossible.

The study also points to a legal "blind spot." Some assistants retrieve content at inference time (the moment a user asks a question). They may then use that data for downstream model improvement (training the AI on new data). Ignoring robots.txt may conflict with laws like the EU AI Act. If an "opt-out" signal is ignored during retrieval, that content may enter a training pipeline. Once there, it may no longer be retractable.

Where the Edges Are

The authors acknowledge several limitations. Because assistants are "black-box" systems, researchers could not identify every internal component. They could not know exactly which sub-tool performed a specific request.

Additionally, the study used a controlled HTML environment. The results might differ on complex, JavaScript-heavy websites. They might also differ on sites with authentication barriers (login requirements). Finally, the study is temporally bounded. AI providers update models and retrieval pipelines frequently. Behaviors observed today may shift tomorrow.

Figures from the paper