Bot vs AI Agent Traffic: Web Bot Auth and Good Bots

Bot controls built to separate humans from scripts now have to make a third decision: whether an automated client is an AI agent acting for a real person, and whether to trust it. Search crawlers, AI training crawlers, user-triggered fetchers and browsing agents all arrive as automated HTTP traffic, yet they carry very different value and risk, and a single "block bots" rule either breaks legitimate agent journeys or lets abuse through. This guide explains how to classify that traffic, how the IETF Web Bot Auth work uses HTTP Message Signatures (RFC 9421) to let bots and agents prove who operates them, what robots.txt (RFC 9309) can and cannot do, and how to build a policy, roll it out and measure it. It is written for application security, edge and fraud teams who already run bot management and now need a deliberate position on AI agent traffic. For the underlying bot defences (scraping, credential stuffing, scalping) see the Bot Management Guide; this page does not repeat them.

Key takeaways

Classify by operator and purpose, not by "bot or human". A training crawler, a search crawler, a user-triggered fetcher and a transacting agent need different policies, even when they come from the same company.
User agents prove nothing on their own. Verify with a cryptographic signature (Web Bot Auth), the operator's published IP ranges, or reverse and forward DNS, and treat an unverified claim to be a known agent as a spoofing signal.
Web Bot Auth is real but still a draft. The IETF webbotauth working group adopted draft-ietf-webbotauth-httpsig-protocol in September 2026; build verification so it can absorb changes to header formats before the RFC is published.
robots.txt expresses preferences, it does not enforce them. User-triggered fetchers and agents generally ignore it by design, so access control for agents belongs at the edge and in the application.
Verified does not mean authorised. A signature tells you who operates the client; rate limits, business logic checks and account-level controls still decide what it may do, especially for intermediary agents that many end users drive.

The automated traffic you now have to tell apart

Operators publish separate identities for separate purposes, and the policy question differs for each. The OpenAI crawler documentation, for example, lists GPTBot (content that may be used for model training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (visits a user starts from ChatGPT or a custom GPT), with an independent robots.txt decision for each and a separate published IP list per agent. Google lists common crawlers such as Googlebot, special-case crawlers such as AdsBot, and user-triggered fetchers including Google-Agent, which it describes as AI agents on Google infrastructure that browse and act on a user's request.

Class	Examples	Who decides when it visits	robots.txt behaviour	Typical policy question
Search crawler	Googlebot, OAI-SearchBot	Operator schedule	Respected	Do we want to be found in this index?
AI training crawler	GPTBot	Operator schedule	Respected (disallow is the training opt-out)	Do we allow content use for model training, and on what terms?
User-triggered fetcher	ChatGPT-User, Google user-triggered fetchers	An end user, in real time	Generally not applied, because a person asked for the page	Do we serve a person who reads us through an assistant?
Browsing or computer-use agent	Google-Agent, agentic browsers	An end user delegating a task	Generally not applied	Can it log in, fill forms, buy or book, and under whose account?
Transacting or commerce agent	Agents using card network agent protocols	An end user with payment credentials	Not relevant	Is the purchase authorised and is the agent trusted by the payment scheme?
Operational bot	Uptime monitors, link checkers, feed fetchers	Contract or configuration	Varies	Is it ours or a supplier's, and is it still needed?
Abusive automation	Scrapers, credential stuffers, scalpers, impersonators	Attacker	Ignored	Detect and stop. See the Bot Management Guide.

Two distinctions matter most. First, direct versus intermediary operation: Cloudflare's bot directory now labels a verified bot as Direct when one narrow operator runs it, or Intermediary when it is an agentic service that many end users operate. Trusting an intermediary's operator is not the same as trusting every user behind it. Second, scheduled versus user-triggered: a crawler that ignores robots.txt is misbehaving, while a user-triggered agent ignoring it is behaving as its operator documents.

How bots and agents prove who they are

There are three verification methods in common use, and they give very different assurance.

Method	How it works	Strength	Weakness
User-agent string	Client declares a token such as `GPTBot` or `Google-Agent`	Free, universal, useful for routing and logs	Trivially spoofed; never a trust signal by itself
Reverse and forward DNS	Reverse-resolve the source IP, check the domain (Google documents googlebot.com, google.com and googleusercontent.com), then forward-resolve and match the original IP	Strong for large operators with dedicated ranges	DNS lookups at request time add latency; not all operators support it; shared cloud egress muddies results
Published IP ranges	Match the source against operator JSON files (for example Google's common-crawlers.json and user-triggered-agents.json, OpenAI's gptbot.json and chatgpt-user.json)	Fast, cacheable, easy to express as a WAF list	Lists drift and must be refreshed; agents running on general cloud infrastructure or residential exits cannot be pinned to ranges
Web Bot Auth (HTTP Message Signatures)	Client signs each request with a private key; the server fetches the operator's public key directory and verifies	Independent of network location; binds the request to a named operator; supports key rotation	Draft specification; header details still changing; verifies the operator, not the end user or the intent

Use them in layers. A request whose user agent claims to be a known crawler but fails IP or signature verification should be scored as impersonation, not treated as unknown. Google's own documentation warns that user-agent strings can be spoofed and points site owners to its IP files and DNS patterns instead.

Web Bot Auth in practice

Web Bot Auth applies RFC 9421 HTTP Message Signatures to automated traffic. RFC 9421 (February 2024) defines the Signature-Input field, which lists the covered components and parameters, and the Signature field, which carries the value. Covered components include derived values such as @method, @authority, @path and @target-uri; parameters include created, expires, nonce, alg, keyid and tag.

Status of the specification

Working group: the IETF Web Bot Auth (webbotauth) working group in the Web and Internet Transport area is chartered to standardise cryptographic authentication of automated clients to websites that mainly serve humans. In scope are search crawlers, archivers, link checkers, AI training crawlers and AI agents acting for end users. Out of scope are HTTP APIs, end-user authentication, non-cryptographic methods, bot intent vocabularies and reputation tracking.
Current draft: draft-ietf-webbotauth-httpsig-protocol-00 (1 September 2026), authored by engineers from Cloudflare and Google, replaced the individual drafts draft-meunier-web-bot-auth-architecture and draft-meunier-webbotauth-httpsig-protocol. It is a working group document, not an RFC, and the charter milestones for sending specifications to the IESG have passed, so expect revisions.
Adoption: Cloudflare verifies Web Bot Auth for its bot directory, AWS WAF Bot Control supports Web Bot Authentication from version 4.0 of the managed rule set, and Google says it is experimenting with the protocol for Google-Agent using the identity https://agent.bot.goog.

What a conforming signed request carries

Element	Requirement in draft-ietf-webbotauth-httpsig-protocol-00	Why it matters to a verifier
`Signature-Agent` header	Structured Field dictionary of https URIs pointing to the operator's key material; member types `directory` (default), `jwks_uri` or `cimd`; must itself be covered by the signature	Tells you where to fetch keys and binds that pointer to the signature
`tag`	`web-bot-auth` on requests; `http-message-signatures-directory` on directory responses	Stops signatures made for other purposes being reused here
Covered components	At least `@authority` or `@target-uri`; `@method` and `@path` recommended	An `@authority`-only signature can be replayed to any path on your host until it expires
`created` and `expires`	Required; expiry no more than 24 hours. Cloudflare advises a very short window, often about a minute	Bounds replay; reject stale or far-future timestamps
`keyid`	base64url JWK SHA-256 thumbprint of the key (RFC 7638, RFC 8037 for Ed25519)	Lets you select the right key from the directory
Key directory	JWKS at `/.well-known/http-message-signatures-directory`, media type `application/http-message-signatures-directory+json`, HTTPS only, only a 200 counts, no automatic redirects	Your verifier fetches attacker-influenced URLs, so it needs SSRF controls
Algorithm	Registered HTTP Message Signatures algorithms; examples use `ed25519` and `rsa-pss-sha512`; shared HMAC secrets must not be used. Cloudflare supports Ed25519 only	Pin an allow-list of algorithms

One example of drift: the IETF -00 draft says signers must send the dictionary form of Signature-Agent and verifiers should still accept the older bare-string form, while Cloudflare's operator documentation at the time of writing describes a quoted string. A verifier that accepts only one form will reject legitimate agents.

Verifier hardening

Fetch keys safely: the draft tells verifiers to cap response size and key count, set fetch timeouts, limit redirects and refuse private, loopback and link-local addresses. Cache directories and refresh on unknown keyid rather than on every request.
Require TLS: origins should refuse signatures sent over unsecured channels.
Allow-list operators: a valid signature from an unknown directory proves only that someone holds a key. Trust comes from your list of operators you have chosen to recognise, or from your edge provider's verified directory.
Plan for rotation and no revocation: operators rotate by publishing the new key beside the old one; the draft defines no revocation, so removal only takes effect when your cache expires. Keep cache lifetimes short enough to matter.
Reject test keys: the draft forbids test keys in production; block any published example keys explicitly.

robots.txt and content-use preferences

RFC 9309 (September 2022) standardised the Robots Exclusion Protocol. Its rules are worth knowing exactly, because many disputes with crawler operators come down to them.

Location and format: /robots.txt at the top level of each host, UTF-8, text/plain. Crawlers should parse at least 500 KiB.
Matching: groups of user-agent, allow and disallow lines; user-agent matching is case-insensitive, path matching case-sensitive; the longest matching rule wins; a crawler with no matching group falls back to *.
Caching and errors: crawlers should not use a cached copy for more than 24 hours. A 4xx response lets a crawler access anything; a 5xx response means assume complete disallow. Serving robots.txt from a misconfigured CDN that returns 404 silently opens the whole site.
Not access control: the RFC is explicit that the protocol is not a security measure, and listing a path in robots.txt makes it discoverable. Never "hide" admin or staging paths there.

Per-agent rules let you separate decisions: OpenAI documents that a site can allow OAI-SearchBot while disallowing GPTBot, and that robots.txt changes take about 24 hours to reach search. For finer content-use signals, the IETF AI Preferences (aipref) working group is standardising a vocabulary for AI-related preferences and ways to attach it through robots.txt, HTTP response headers and well-known URIs. Its charter excludes enforcement and crawler authentication, which is why Web Bot Auth and aipref are complementary: one says who is asking, the other says what you prefer they do.

Building the traffic policy

Write the policy as a matrix of verified identity, purpose and path sensitivity, then express it in edge rules. Managed platforms already expose the building blocks: Cloudflare classifies verified bots by behaviours such as Search, Agent, Training and Transact, and AWS WAF Bot Control attaches labels such as awswaf:managed:aws:bot-control:bot:verified that custom rules can match.

Traffic	Public content	Logged-in journeys	High-value actions (checkout, transfer, account change)
Verified search crawler	Allow, crawl-rate limits	Not applicable	Not applicable
Verified training crawler	Your content-licensing decision (allow, block, or charge)	Block	Block
Verified user-triggered fetcher or agent (direct)	Allow with per-operator rate limits	Allow only through the user's own session, with normal authentication	Step-up or explicit user confirmation
Verified intermediary agent	Allow with per-operator and per-session limits	Treat the end user as the principal; apply account-level risk scoring	Step-up, delegated authorisation or agent payment protocol checks
Unverified client claiming to be a known agent	Challenge or block	Block	Block and alert
Unknown automation	Normal bot management scoring	Normal bot and fraud controls	Normal bot and fraud controls

Decision guidance

Publisher or content site: the main choice is content use. Allow search crawlers, decide on training crawlers commercially, and serve user-triggered fetchers so you stay visible in AI answers. Some edges now support charging: AWS WAF documents a Monetize action that returns HTTP 402 Payment Required for CloudFront-attached web ACLs.
E-commerce, travel or ticketing: agents that buy are the question. Card schemes are publishing agent trust signals; Visa describes its Trusted Agent Protocol as carrying agent intent, optional consumer recognition and payment information in merchant-specific, time-bound signatures that cannot be replayed. See the Agent Payments Protocols Guide and the Agentic Commerce Identity Guide.
Financial services and account-centric apps: keep agents on the same authentication path as the user. Verified operator identity can lower friction on public pages, but never substitutes for the customer's own credentials or step-up on payments.
Small organisations: use the managed verified-bot categories and AI bot presets your CDN or WAF provides, plus per-agent robots.txt lines. Do not build your own signature verifier.
Large organisations: run your own operator allow-list and verification telemetry on top of the edge provider's, so a provider's classification change cannot silently change your exposure, and so the same policy holds across multiple CDNs.

Where identity fits

Your own bots and agents are non-human identities on someone else's site. If you operate crawlers, monitors or customer-facing agents, the Web Bot Auth private key is a production signing credential: keep it in a vault or KMS, give each agent its own key so one leak does not taint all traffic, rotate by overlap as the draft describes, and publish only public parameters in the directory. See the Agentic AI Identity Guide and the Agent-to-Agent Authentication Guide.

Rolling it out

Inventory current automated traffic. Pull 30 days of edge logs, group by declared user agent, verification result and path class.
Agree the policy matrix with content, legal, fraud and product owners. Training-crawler access is a commercial and legal decision, not a security default.
Turn on verification in log-only mode. Enable verified-bot labelling (or Web Bot Auth verification on AWS WAF with the rule set version selected explicitly, since it is not the default) and record outcomes without blocking. AWS also documents a test mode for monetisation rules.
Update robots.txt per agent, test it with each operator's documented tokens, and confirm the file returns 200, not 4xx or 5xx, from every edge and host.
Enforce on the impersonation case first: block or challenge clients that claim a known identity and fail verification. This has the lowest false-positive risk and the highest abuse value.
Apply per-operator rate limits and path rules for verified agents, starting with high-value actions. Pilot on one property or one region.
Roll back cleanly: keep each change as a separately toggled rule with an owner, so a broken agent journey can be reverted to log-only without disabling the rest of bot management.
Review quarterly: new agent identities appear constantly, operators retire tokens (Google renamed Google-NotebookLM to Google-GeminiNotebook), and IP files change.

Detection and measurement

Log, per request: declared user agent, verification method and result (signature valid, IP match, DNS match, failed, none), Signature-Agent directory URL, keyid, signature age, operator category, path class and action taken. Alert on spikes in failed verifications for a single claimed identity, on unknown directories presenting valid signatures, and on verified agents reaching sensitive paths they have never touched before.

Metric	What it shows	Healthy direction
Share of automated traffic that is cryptographically or IP verified	How much of your agent policy rests on proof rather than claims	Rising
Impersonation rate: requests claiming a known identity that fail verification	Spoofing pressure and rule effectiveness	Detected and blocked, trend watched
False-positive rate on verified agent journeys (user complaints, failed agent checkouts)	Whether you are breaking legitimate delegated use	Falling
Signature verification failures by reason (expired, unknown key, malformed)	Interoperability problems versus attacks	Stable and explained
Time to add or revoke an operator in policy	Operational agility as agents change	Under one working day
Origin load and cost attributable to each verified operator	Whether rate limits and licensing terms match reality	Within agreed budgets

Framework mapping

Framework	Reference	How it applies
NIST CSF 2.0	PR.AA-03; DE.CM-01; DE.AE-02	Authenticate services (verified bots and agents); monitor network services; analyse anomalous automated activity
NIST SP 800-53 Rev. 5	IA-9 Service Identification and Authentication; IA-8 Identification and Authentication (Non-organizational Users)	Verify the operator of an automated client before trusting it; agents acting for customers are non-organisational users
NIST SP 800-53 Rev. 5	AC-3 Access Enforcement; SC-5 Denial-of-service Protection; SC-7 Boundary Protection	Path and action rules per traffic class; rate limits; edge enforcement
NIST SP 800-53 Rev. 5	AU-2 Event Logging; SI-4 System Monitoring	Verification outcomes and agent identity in logs; alerting on impersonation
MITRE ATT&CK v20	T1595 Active Scanning; T1110.004 Credential Stuffing; T1499.002 Service Exhaustion Flood	Abusive automation that hides among, or impersonates, legitimate crawlers and agents

Common failure patterns

Allow-listing by user agent. Rules that exempt anything named like a well-known crawler or AI assistant from bot checks give attackers a free pass. Verify, then exempt.
Blanket "block AI" rules. Blocking every AI-labelled client also blocks user-triggered fetchers your customers rely on, and removes you from AI search answers, while scrapers that never declare themselves carry on.
Treating a valid signature as authorisation. A verified intermediary agent may be driven by thousands of users, some abusive. Agent swarms can mass-create accounts and hunt credentials, as the GemStuffer RubyGems campaign shows; account-creation and abuse controls must still apply.
Relying on robots.txt to keep agents out. User-triggered agents generally ignore it by design, and a 4xx on the file means "everything allowed". Use real access controls on anything sensitive.
Accepting long-lived or path-blind signatures. Signatures covering only @authority with long expiry windows can be captured and replayed against other paths. Enforce short expiry and recommend method and path coverage.
Ignoring your own agents' outbound behaviour. If agents you operate misbehave on other sites, your signing identity is what gets blocked. Rogue agent activity against government sites, as in the OpenAI agents and US government websites incident, shows why agent operators need egress controls and attributable identities.

Practitioner checklist

Classify automated traffic into search, training, user-triggered, browsing agent, transacting agent, operational and abusive, and give each an owner.
Verify claimed identities with signatures, published IP ranges or reverse and forward DNS; never trust the user agent alone.
Score failed verification of a known identity as impersonation and block or challenge it first.
Enable Web Bot Auth verification through your edge provider, or build a verifier with SSRF controls, algorithm allow-lists and tolerance for both Signature-Agent forms.
Enforce short signature lifetimes and prefer signatures covering method and path.
Publish per-agent robots.txt rules, confirm the file returns 200 everywhere, and track aipref drafts for content-use signals.
Keep agents on the user's own authentication path for logged-in and high-value actions.
Protect any signing keys your own agents use as production credentials, one key per agent.
Log verification outcomes and report the six metrics above to the bot and fraud owners monthly.
Review operator lists, IP files and draft changes every quarter.

Bot vs AI Agent Traffic Management Guide: Web Bot Auth and Good Bots