AI Crawler and Scraper Controls: Managing Bots That Train on Your Site

Most sites that try to "block AI" either block nothing, because they named the wrong user agent, or block too much, because they cut off the search crawlers that send them visitors. AI companies now run separate agents for model training, for search indexing and for fetching a page when a user asks a question, and each obeys robots.txt differently. This guide covers how to tell those agents apart, how to write robots.txt rules and the newer IETF AI usage preferences that express what you actually want, what llms.txt does and does not do, how pay per crawl works, how to verify that a crawler is who it claims to be, and how to enforce and monitor your policy. It is written for web, platform and security teams and for the publishers and executives who set content licensing policy. Abuse bots such as credential stuffers are covered in the Bot Management Guide, and the robots.txt protocol basics in the Bot vs AI Agent Traffic Management Guide.

Key takeaways

• Decide per purpose, not per company. Training, search indexing and user-triggered fetching are separate agents at most AI operators, and blocking one does not block the others.
• Some controls are tokens, not crawlers. Google-Extended and Applebot-Extended never appear in your logs; they only tell the main crawler how its data may be used, so blocking them does not remove you from search.
• User-triggered fetchers may ignore robots.txt. OpenAI and Perplexity both document that agents acting for a user may not follow robots.txt, so those need edge rules if you must stop them.
• A user agent string is a claim, not an identity. Verify crawlers against the IP ranges and reverse DNS the operators publish before you allow, block or bill them.
• robots.txt expresses preference, not permission. RFC 9309 states its rules are not access authorisation; enforcement needs your edge, WAF or bot controls, and licensing needs contracts.

The three kinds of AI agent hitting your site

Before writing any rule, separate what each agent does with your content. The distinction matters commercially: blocking training costs you little traffic, blocking AI search can cost you referrals, and blocking user fetches can break features such as link previews in assistants.

Crawler and token reference by operator

Operators are listed alphabetically; the order is not a ranking. Each row is taken from the operator's own documentation, checked on 8 November 2026. Names change, so recheck these pages each quarter.

Behaviour notes from the same documents, one per operator:

• Anthropic: its bots honour robots.txt, support the non-standard Crawl-delay directive and do not bypass CAPTCHAs. Rules must be set on each subdomain, and Anthropic warns that IP blocking may not reliably opt you out. From 30 October 2026 it also recognises a machine-readable "no AI training" signal based on an IETF aipref draft, with rules that name ClaudeBot taking priority.
• Apple: Applebot follows Googlebot rules when a file has no Applebot group, does not follow Crawl-delay, and disallowing Applebot-Extended does not remove pages from Apple's search features.
• Common Crawl: blocked with a User-agent: CCBot group; it warns that other crawlers falsely claim to be CCBot, and reverse DNS is not yet available for its IPv6 ranges.
• Google: Google-Extended controls use for Gemini training and grounding, and Google states it does not affect inclusion or ranking in Google Search. Google says it intends to support the AIPREF shared training preference signal, as a __usage_ai-training user agent group, in early 2027.
• OpenAI: disallowing GPTBot signals content should not be used for training; OAI-SearchBot changes take about 24 hours to reach ChatGPT search; ChatGPT-User acts for users and robots.txt rules may not apply.
• Perplexity: PerplexityBot powers search results; Perplexity-User generally ignores robots.txt because a user triggers it; changes can take up to 24 hours, and Perplexity recommends using its JSON files as the source of truth for WAF rules.

Writing the robots.txt policy

RFC 9309 matches user agent groups case-insensitively and uses the most specific path rule, so a dedicated group for an agent overrides the * group entirely for that agent. A common publisher policy, described rather than prescribed: allow search crawlers (including AI search agents), disallow training agents and tokens, and leave * for everything else. In practice that is a group per training agent or token (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) with Disallow: /, and no restriction on Googlebot, Applebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot.

Decision guide

Small sites can stop at robots.txt and a quarterly check of operator pages. Large publishers and e-commerce sites with valuable catalogues usually need the full stack: robots.txt, verified-bot rules at the edge, monitoring and a licensing position agreed with legal.

AI usage preferences: the IETF aipref drafts

robots.txt can only say "fetch" or "do not fetch". The IETF AI Preferences (aipref) working group is standardising a way to say what may be done with content that is fetched. As checked on 8 November 2026, both core documents are Internet-Drafts, not RFCs: draft-ietf-aipref-vocab-08 and draft-ietf-aipref-attach-05. Details may change before publication, so treat syntax as provisional.

• Vocabulary: three categories, train-ai (changing a generative model's parameters), ai-use (using content as model input when the user did not supply it) and search (finding content and sending users to it with a direct link), each set to y or n. Missing values mean unknown.
• Attachment: a Content-Usage HTTP response header, for example Content-Usage: train-ai=n, and a Content-Usage rule inside a robots.txt user agent group, with an optional path so one directory can carry a different preference.
• Adoption: Anthropic says it recognises a "no AI training" signal based on an aipref draft from 30 October 2026, and Google says it intends to support the training preference signal in early 2027. Adding the rule now costs little, but keep the per-agent groups as well, because named-agent rules take priority and not every operator reads the new signal.

llms.txt: what it is and is not

llms.txt is a proposal, first published in September 2024 and now at version 2 on llmstxt.org, for a Markdown file at /llms.txt (or a subpath such as /docs/llms.txt) that gives AI agents a curated map of a site. The only required element is an H1 with the site name; an optional blockquote summary and H2 sections of links follow, with an "Optional" section for secondary links. It also suggests serving clean Markdown copies of pages at the same URL with .md appended.

• It is not a blocking control. The proposal positions robots.txt as the file that says which access is acceptable, while llms.txt is read on demand by agents that are already allowed.
• Use it where you want AI visibility, for example product documentation or API references you want assistants to answer from correctly.
• Do not list anything sensitive. Like robots.txt, it is public and makes listed paths easy to find.
• Keep it consistent with robots.txt. Linking paths in llms.txt that robots.txt disallows sends mixed signals to compliant agents.

Pay per crawl and licensing

Pay per crawl uses HTTP status 402 Payment Required to turn crawling into a transaction. In the implementation documented by one CDN provider (Cloudflare, cited below), a site owner sets a price per zone; a crawler either signals payment intent in request headers and receives the page with HTTP 200, or receives a 402 response with pricing. In that implementation the provider acts as merchant of record, the feature was in closed beta when checked, and WAF or bot management blocks take precedence over a charge. The model depends on knowing which crawler is asking, which is why it relies on verified bot identity such as signed requests; see Web Bot Auth and HTTP Message Signatures for Agents.

Direct licensing deals with AI companies are the other route. Whatever the commercial model, write the agreement around verifiable identifiers (named user agents plus the IP ranges or signing keys they use), the purposes allowed (training, search, user fetch), attribution and link requirements, and a right to audit. Copyright and provenance questions on the AI side are covered in the AI Training Data Provenance and Copyright Exposure Guide.

Verifying crawlers and enforcing at the edge

Anyone can send User-Agent: GPTBot. Before a crawler gets an allow rule, an exemption from rate limits or a bill, confirm it:

1. Load operator IP ranges automatically. Fetch each operator's published JSON file on a schedule (daily is reasonable) into an edge IP list, and alert if a fetch fails or the list empties.
2. Use forward-confirmed reverse DNS where offered. Resolve the IP to a hostname, check the domain (for example googlebot.com, applebot.apple.com or crawl.commoncrawl.org), then resolve the hostname back and confirm it matches.
3. Treat mismatches as impostors. A request claiming a known AI user agent from an unlisted IP is scraping under a borrowed name; rate limit or challenge it under your bot management policy, see the Web Scraping Defence Guide.
4. Write edge rules for agents robots.txt cannot stop. If policy says no user-triggered AI fetches, block the documented user agents and their verified ranges at the WAF, and accept that this also stops users of those assistants from reading your pages through them.
5. Prefer cryptographic identity as it arrives. Signed HTTP requests let you verify an agent without maintaining IP lists.
6. Roll out in monitor mode first. Log matches for one to two weeks, confirm no search crawler or partner is caught, then enforce. Roll back by switching rules to log only.

Detection and measurement

Add the user agent, verified-bot result, ASN and response status to web and CDN logs, then track:

• AI crawler share: requests and bytes from each verified AI agent per week, split by training, search and user-triggered.
• Impostor rate: requests claiming an AI user agent that fail IP or DNS verification.
• Compliance rate: requests from verified agents to paths their robots.txt group disallows; anything above zero for an agent that documents compliance is worth raising with the operator.
• Referral return: visits referred from AI search products, to judge what allowing search agents is worth.
• robots.txt availability: uptime and status codes for /robots.txt on every host, because a 5xx tells compliant crawlers to stop entirely and a 4xx tells them everything is allowed.

Framework mapping

Common failure patterns

• Blocking Googlebot or Applebot to stop AI training, which removes the site from search; Google-Extended and Applebot-Extended exist so you do not have to.
• Expecting robots.txt to stop user-triggered fetchers, when two operators document that it may not apply to them.
• Allowlisting by user agent alone, which lets any scraper through by copying a crawler's name; verification has to be by IP, DNS or signature.
• A robots.txt that errors, such as a CDN returning 404 or 500 for the file on one host, silently changing policy for every crawler.
• Forgetting subdomains and new hosts: robots.txt is per host, so documentation, shop and media subdomains need their own files.
• Treating llms.txt or aipref as enforcement and skipping edge controls and contracts for content that matters commercially.
• Setting the policy once. Operators add and rename agents; a list written in 2024 misses agents documented since.

Practitioner checklist

• Agree a written content use policy covering training, AI search and user-triggered fetches, signed off by legal and the content owner.
• List every host and subdomain and confirm each serves a valid /robots.txt with a 200 response.
• Add per-agent groups for training agents and usage tokens, and keep search agents allowed if visibility matters.
• Add an aipref Content-Usage preference alongside the named groups, noting it is still an Internet-Draft.
• Publish llms.txt only for content you want assistants to use, and keep it consistent with robots.txt.
• Load operator IP range files into the edge daily and verify claimed AI crawlers before allowing or billing them.
• Write WAF rules for user-triggered fetchers only if policy requires it, and test in log-only mode first.
• Report monthly on AI crawler share, impostor rate, compliance rate and AI search referrals.
• Recheck each operator's crawler documentation every quarter.

Standards and references

• IETF: RFC 9309, Robots Exclusion Protocol
• IETF: A Vocabulary For Expressing AI Usage Preferences (Internet-Draft)
• IETF: Associating AI Usage Preferences with Content in HTTP (Internet-Draft)
• IETF: AI Preferences (aipref) working group documents
• llmstxt.org: The /llms.txt file proposal
• Cloudflare Docs: What is pay per crawl

Related NHI Mgmt Group resources: Bot Management Guide · Bot vs AI Agent Traffic Management Guide · Web Bot Auth and HTTP Message Signatures for Agents · Web Scraping Defence Guide