Every AI operator now publishes its own set of crawler names, robots.txt tokens and page-level tags, and the same words mean different things at different companies. This tracker lists, from each operator's own documentation, the controls a site owner can use to allow or refuse AI training, search indexing, user-triggered fetching and grounding (using a page as input to a generated answer). Every operator gets the same fields, and where an operator's documentation does not state something, the cell says so. It is written for web, SEO, legal and security teams who need the exact token or tag and the date it was checked. Operators are listed alphabetically; order is never a ranking.
Last checked: 2026-11-15.
This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.
Key takeaways
- Four purposes, four controls. Training, search indexing, user-triggered fetching and grounding are separate decisions, and most operators give each its own agent, token or tag.
- Grounding now has page-level tags. Microsoft Bing announced ai-training and ai-grounding preferences on 30 October 2026, set by meta tag or X-Robots-Tag, taking effect as pages are crawled from 1 February 2027.
- A shared training token is arriving. Bing documents the user agent group __usage_ai-training, Google says it intends to support it in early 2027, and Anthropic began recognising an IETF aipref "no AI training" signal on 30 October 2026.
- Snippet rules reach AI answers. Google documents that nosnippet stops content being a direct input to AI Overviews and AI Mode, and Apple documents that nosnippet excludes a page from AI context.
- User-triggered fetchers differ. OpenAI, Meta and Perplexity document that their user-initiated fetchers may not follow robots.txt; Anthropic documents that its bots honour it.
What does this tracker cover?
The tracker covers controls a site owner sets on their own site: robots.txt groups, robots meta tags, X-Robots-Tag HTTP headers and, where documented, webmaster tool settings. It covers eight operators whose documentation was fetched on 15 November 2026: Anthropic, Apple, Common Crawl, Google, Meta, Microsoft, OpenAI and Perplexity. It does not cover edge enforcement, IP verification or pay per crawl in depth; those are in the AI Crawler and Scraper Controls guide, with protocol basics in the Bot vs AI Agent Traffic Management Guide.
Terms used on this page:
- Crawler: an agent that visits pages automatically and identifies itself with a user agent string.
- Token: a name used in a robots.txt User-agent line that may not match any crawler, for example Google-Extended or Applebot-Extended, which only tell an existing crawler how its data may be used.
- User-triggered fetcher: an agent that visits a page because a person asked an AI product to read it.
- Grounding: Bing's definition is using content as input to help inform AI-generated responses; indexing and ranking for traditional search results is not grounding.
Which robots.txt tokens does each operator document?
As of 15 November 2026. Each row comes from the operator's own crawler page listed in Standards and references.
| Operator | Training | Search or index | User-triggered fetch | Other documented agents |
|---|---|---|---|---|
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | Not stated by Anthropic |
| Apple | Applebot-Extended (token; does not crawl) | Applebot (Spotlight, Siri, Safari) | Not stated by Apple | iTMS (Apple Podcasts registered content) |
| Common Crawl | CCBot (open web dataset that others reuse) | Not stated by Common Crawl | Not stated by Common Crawl | Not stated by Common Crawl |
| Google-Extended (token; no separate user agent) | Googlebot | Not stated on the common crawlers page | Google-CloudVertexBot (owner-requested Vertex AI Agents crawls), GoogleOther | |
| Meta | meta-externalagent | meta-webindexer (Meta AI search) | meta-externalfetcher | facebookexternalhit (link previews), meta-externalads |
| Microsoft | __usage_ai-training (shared token, robots.txt) | Bingbot | Not stated in the Bing announcement | Not stated in the Bing announcement |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | OAI-AdsBot (submitted ad landing pages) |
| Perplexity | Not used for foundation model training, per Perplexity | PerplexityBot | Perplexity-User | Not stated by Perplexity |
Purpose labels follow each operator's wording. Some agents serve more than one purpose: Apple states Applebot data may help train Apple foundation models and supply context for AI answers, Meta states meta-externalagent crawls both for training and for indexing to improve products, and OpenAI states that if both GPTBot and OAI-SearchBot are allowed it may use one crawl's results for both purposes.
How does each operator treat robots.txt and changes?
| Operator | robots.txt behaviour documented | Time for changes to apply | Published IP or DNS verification |
|---|---|---|---|
| Anthropic | Bots honour robots.txt; supports non-standard Crawl-delay; rules needed on each subdomain | Not stated by Anthropic | claude.com/crawling/bots.json |
| Apple | Follows Applebot rules, or Googlebot rules if no Applebot group exists; ignores Crawl-delay; iTMS does not follow robots.txt | Not stated by Apple | Reverse DNS in applebot.apple.com, plus applebot.json |
| Common Crawl | Blocked with a User-agent: CCBot group; also offers an Opt-Out Registry | Not stated by Common Crawl | Reverse DNS in crawl.commoncrawl.org (not yet for IPv6), plus index.commoncrawl.org/ccbot.json |
| Google-Extended groups govern use, not crawling; Google-CloudVertexBot groups also match Googlebot | Not stated on the common crawlers page | Not stated on the common crawlers page | |
| Meta | meta-externalagent, meta-webindexer and meta-externalads follow disallow rules; facebookexternalhit and meta-externalfetcher may bypass robots.txt | Up to 24 hours | Not stated on the fetched page |
| Microsoft | Training preference by __usage_ai-training group; an Allow preference does not override a robots.txt block on Bingbot | On recrawl; IndexNow may make a recrawl more likely | Not stated in the Bing announcement |
| OpenAI | Each bot set independently; ChatGPT-User acts for users and robots.txt may not apply | About 24 hours for search changes | gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json |
| Perplexity | PerplexityBot honours robots.txt; Perplexity-User generally ignores it because a user requested the fetch | Up to 24 hours | perplexitybot.json, perplexity-user.json |
Which page-level tags and headers control AI use?
robots.txt works at site or path level. Page-level controls go in a robots meta tag or the X-Robots-Tag header, and only some operators document AI effects for them. As of 15 November 2026:
| Operator | Page-level control | Example syntax | Documented AI effect |
|---|---|---|---|
| Anthropic | Not stated by Anthropic | Not stated by Anthropic | Not stated by Anthropic |
| Apple | noindex, nosnippet, nofollow, none, all; applebot meta name; X-Robots-Tag; schema.org isAccessibleForFree | X-Robots-Tag: applebot: nosnippet | nosnippet excludes the page from AI context; paywalled pages marked with isAccessibleForFree are excluded from AI context |
| Common Crawl | Not stated by Common Crawl | Not stated by Common Crawl | Not stated by Common Crawl |
| nosnippet, max-snippet, data-nosnippet, noindex; googlebot meta name; X-Robots-Tag | <meta name="googlebot" content="nosnippet"> | nosnippet prevents use as a direct input for AI Overviews and AI Mode; max-snippet limits how much may be used | |
| Meta | Not stated on the fetched page | Not stated on the fetched page | Not stated on the fetched page |
| Microsoft | ai-training and ai-grounding values on the Bingbot meta name or X-Robots-Tag (from 1 February 2027) | <meta name="Bingbot" content="ai-grounding:disallow"> | disallow: not used to inform generated responses; disallowextended: Microsoft-operated experiences only; allow: eligible (default) |
| OpenAI | noindex | <meta name="robots" content="noindex"> | Keeps a page from surfacing in ChatGPT search; OAI-SearchBot must be allowed to crawl the page to read the tag |
| Perplexity | Not stated by Perplexity | Not stated by Perplexity | Not stated by Perplexity |
Google notes that the noarchive rule is no longer used by Google Search because the cached link feature no longer exists, and that only the googlebot and googlebot-news names are supported in its robots meta tag. Where rules conflict, Google applies the more restrictive one.
What are Bing's ai-training and ai-grounding preferences?
Microsoft Bing published the preferences on 30 October 2026 in the post "Your content, your choice: clearer preferences for training and grounding". They govern how content is used, not whether Bingbot may crawl it.
- AI training: a shared industry preference. Set it for a site or path in robots.txt with a
User-Agent: __usage_ai-traininggroup, or per page withai-training:disallowin the Bingbot meta tag orX-Robots-Tag: Bingbot: ai-training:disallow. Page-level settings override robots.txt. The default is allow. - AI grounding: Microsoft-specific and page level only, with values
allow(default),disallowextendedanddisallow. There is no site-wide robots.txt grounding preference. - disallowextended: limits grounding to Microsoft-operated experiences, which the post lists as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes third-party grounding services such as Web IQ and the grounding API in Foundry.
- Search eligibility: the preferences do not change eligibility for traditional Bing results, and the current Bing search syndication API is not affected by the grounding preference.
- Activation: a preference takes effect only once Microsoft crawls the page and discovers it.
| Date | Event (as announced 30 October 2026) |
|---|---|
| 15 December 2026 | Temporary grounding transition preferences open in Bing Webmaster Tools, only for sites using NoArchive or NoCache on that date |
| 1 February 2027 | New preferences take effect as pages are crawled |
| 1 March 2027 | Deadline for page-level preferences and temporary grounding preferences; NoArchive and NoCache stop being read as training or grounding preferences |
| 30 September 2027 | Temporary preferences expire (or earlier, at the page's next recrawl) |
Pages with no recorded temporary preference by 1 March 2027 default to allow for grounding until recrawled, and there is no temporary training setting. These are announced dates, so confirm them against the Bing post and the Bing Webmaster Tools help FAQ before acting. How Copilot uses grounding is covered in Microsoft Copilot Web Answers Explained.
What is the shared AI training preference signal?
The IETF AI Preferences (aipref) working group is drafting a vocabulary for expressing AI usage preferences and a way to attach it to content. As shown on the working group's documents page on 15 November 2026, draft-ietf-aipref-vocab is at version 08 (13 September 2026) and draft-ietf-aipref-attach at version 05 (18 August 2026), both Internet-Drafts intended as Proposed Standard. These are drafts and may change.
| Operator | Position stated in its documentation |
|---|---|
| Anthropic | Recognises a machine-readable "no AI training" signal from 30 October 2026, following draft-kelley-aipref-training-preference-signal; robots.txt rules naming ClaudeBot take priority for the affected paths; interpretation may change as standards evolve |
| Apple | Not stated by Apple |
| Common Crawl | Not stated by Common Crawl |
Intends to support the AIPREF training preference as user-agent: __usage_ai-training in early 2027 | |
| Meta | Not stated on the fetched page |
| Microsoft | Supports __usage_ai-training in robots.txt and ai-training at page level, effective as pages are crawled from 1 February 2027 |
| OpenAI | Not stated by OpenAI |
| Perplexity | Not stated by Perplexity |
Policy design around aipref, including how it relates to named crawler groups, is covered in the AI Crawler and Scraper Controls guide.
Which control matches which goal?
Each row lists what the operators' documentation supports for that goal. Operators are listed alphabetically within each cell.
| Goal | Documented controls | What it does not do |
|---|---|---|
| Refuse model training, stay in search | Disallow Applebot-Extended, CCBot, ClaudeBot, Google-Extended, GPTBot, meta-externalagent; __usage_ai-training for Bing (and Google from early 2027) | Google-Extended does not affect Search inclusion or ranking; Applebot-Extended does not remove pages from Apple search features |
| Stay out of AI answers in Google Search | nosnippet or max-snippet on the page, visible to Googlebot | Also changes classic snippets; Google documents Google-Extended for Gemini Apps and Vertex AI, not for Search features |
| Stay out of Copilot and other Bing grounding | ai-grounding:disallow, or disallowextended to allow Microsoft-operated experiences only | Does not affect traditional Bing results |
| Stay out of AI search answers at other engines | Disallow Claude-SearchBot, meta-webindexer, OAI-SearchBot, PerplexityBot; OpenAI also honours noindex | OpenAI notes opted-out sites can still appear as navigational links |
| Stop user-triggered fetches | Disallow Claude-User (Anthropic documents its bots honour robots.txt) | ChatGPT-User, meta-externalfetcher and Perplexity-User may not follow robots.txt; edge rules are needed |
For example, a site that wants to refuse training by OpenAI but stay in ChatGPT search writes two separate groups:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /How should teams roll out and monitor these controls?
- Write the policy per purpose. Record for each operator whether training, search, user fetching and grounding are allowed, and who owns the decision (usually legal or editorial, not only engineering).
- Change robots.txt in one reviewed commit. Keep the previous file so rollback is a single revert, and remember OpenAI and Perplexity document up to about 24 hours before changes apply.
- Apply page tags through templates. Put nosnippet, max-snippet and Bing ai-grounding values in page templates or response headers rather than by hand, and confirm Googlebot can see them with Google's URL Inspection tool.
- Plan the Bing transition. If the site uses NoArchive or NoCache today, decide before 1 March 2027 whether to publish ai-training and ai-grounding values or record temporary preferences in Bing Webmaster Tools from 15 December 2026.
- Monitor logs by user agent. Count requests per documented agent, verify claimed agents against the published IP files, and alert when a disallowed training agent keeps fetching.
- Measure the effect. Track referrals from AI answers (ChatGPT referrals carry utm_source=chatgpt.com) before and after each change, using the tools in the AI Search Citation Reporting Tools Tracker.
Useful measures: share of documented AI agents with an explicit decision in robots.txt; percentage of requests claiming an AI user agent that verify against published ranges; time from policy change to observed crawler behaviour; AI referral sessions per month; number of templates carrying the agreed page-level tags.
Common failure patterns
- Blocking search to stop training: disallowing OAI-SearchBot, PerplexityBot or Googlebot when the intent was only to refuse training removes the site from those answers.
- Treating tokens as crawlers: searching logs for Google-Extended or Applebot-Extended finds nothing, because they never fetch pages.
- noindex behind a block: OpenAI documents that OAI-SearchBot must be able to crawl a page to read its noindex tag, so a robots.txt block can stop the tag being seen.
- Relying on NoArchive or NoCache for Bing: after 1 March 2027 Bing no longer reads them as training or grounding preferences.
- Expecting robots.txt to stop user fetches: three operators document that their user-triggered fetchers may not follow it.
Practitioner checklist
- List every operator in this tracker and record a decision for training, search, user fetching and grounding.
- Check robots.txt groups use the exact documented tokens, on every subdomain.
- Decide on nosnippet or max-snippet for pages that should not feed Google AI Overviews and AI Mode.
- Plan Bing ai-training and ai-grounding values before 1 February 2027 and replace NoArchive or NoCache reliance before 1 March 2027.
- Add __usage_ai-training to the policy review for early 2027, when Google says it intends to support it.
- Verify claimed AI crawlers against published IP files before allowing or blocking them at the edge.
- Recheck each operator's crawler page quarterly and date every change in your own records.
Standards and references
- Microsoft Bing Blog: Your content, your choice: clearer preferences for training and grounding
- Google Search Central: Google's common crawlers
- Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag specifications
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Apple: About Applebot
- Meta: Meta Web Crawlers
- Perplexity: Perplexity crawlers
- Common Crawl: CCBot
- IETF: AI Preferences (aipref) working group documents
Related NHI Mgmt Group resources: AI Crawler and Scraper Controls, AI Answer Engines Hub, What Is Generative Engine Optimisation (GEO)? Definition, Research and Limits, Bot vs AI Agent Traffic Management Guide.