Websites cannot tell a real AI agent or crawler from an impostor using the same user agent string, and IP allow lists break as soon as agents run from shared cloud ranges. Web Bot Auth answers this with cryptography: the agent signs each HTTP request with a private key under RFC 9421 HTTP Message Signatures, publishes the public keys in a well-known directory, and the site verifies the signature before deciding how to treat the request. The IETF webbotauth working group adopted its protocol document in September 2026, replacing the earlier individual architecture draft, and CDNs already verify signed traffic. This guide gives the implementation detail for both sides: what an agent builder must sign and publish, how a publisher or edge team should verify, cache and score, and which parts are still moving. It is for agent platform engineers, edge and bot management teams, and identity architects. Statuses were checked on the IETF datatracker on 27 October 2026. For classifying agent traffic and building access policy, see the Bot vs AI Agent Traffic Management Guide; this page does not repeat it.
Key takeaways
- The current document is draft-ietf-webbotauth-httpsig-protocol-00. It replaces draft-meunier-webbotauth-httpsig-protocol, which itself replaced the expired draft-meunier-web-bot-auth-architecture. Cite and build against the WG draft and expect changes.
- Three moving parts: a
Signature-Agentheader pointing at the operator's key source, an RFC 9421 signature taggedweb-bot-authcovering at least the authority and that header, and a JWKS directory at/.well-known/http-message-signatures-directory. - Signatures prove the operator, not the permission. The draft states that a valid signature does not show the operator is honest or the request authorised; authorisation, delegation and revocation are out of scope.
- Verifiers fetch URLs supplied by strangers. Directory discovery is an SSRF and resource exhaustion surface; the draft asks verifiers to limit size, key count, time and redirects and to block private address ranges.
- Short signatures, overlapping keys. Generate a fresh signature per request with a short
expires, rotate keys by publishing the new key alongside the old, and remember that removing a key is not instant revocation.
Document status and lineage
| Document | Version and date | Status | Role |
|---|---|---|---|
| RFC 9421 HTTP Message Signatures | February 2024 | Proposed Standard | The signing mechanism everything builds on |
| draft-ietf-webbotauth-httpsig-protocol | -00, 1 September 2026 | WG document; header says Standards Track | Signature-Agent header, key directory format and well-known URI, verification rules |
| draft-meunier-web-bot-auth-architecture | -05, 2 March 2026 | Replaced | Original architecture; do not cite for current behaviour |
| draft-meunier-http-message-signatures-directory | -05, 2 March 2026 | Replaced by draft-meunier-webbotauth-httpsig-directory | Original directory specification |
| draft-meunier-webbotauth-registry | -03, 26 June 2026 | Individual draft | Signature Agent Cards and registries of agents |
| draft-rescorla-anonymous-webbotauth | -01, 19 July 2026 | Individual draft | Anonymous bot authentication for authorisation and rate limiting |
| draft-illyes-webbotauth-cbcp, draft-illyes-webbotauth-jafar, draft-kelley-webbotauth-sail | -01, -01, -00, 22 October 2026 | Individual drafts | Crawler best practice, JSON format for bot IP ranges, source address identity lists |
The working group charter covers search crawlers, archivers, link checkers, AI training crawlers and AI agents acting for end users against sites that mainly serve humans. HTTP APIs, agent-to-agent interfaces, end-user authentication and bot reputation are out of scope. The charter milestones targeted April 2026 for sending the authentication specification to the IESG; the protocol document is still at -00, so plan for further revisions.
What the agent sends
| Element | Requirement in the WG draft | Notes for implementers |
|---|---|---|
Signature-Agent header | Dictionary structured field whose values are https URIs; optional type parameter: directory (default), jwks_uri or cimd | Points the verifier at the operator's keys. Cloudflare's current documentation shows a quoted string form; check what each verifier accepts while formats converge. |
| Covered components | Must cover @authority or @target-uri, plus the signature-agent header | Covering @method, @path or @query-param is recommended to narrow what a captured signature can be reused for |
created and expires | Required; the draft recommends expires no more than 24 hours after creation | Use far shorter windows in practice. Cloudflare recommends about a minute because it does not check nonces. |
keyid | Required; base64url JWK SHA-256 thumbprint of the key | Lets verifiers find the key in the directory |
tag | Required; value web-bot-auth | Verifiers may ignore signatures with other tags |
nonce | Follows RFC 9421 replay guidance | Recommended; random per request |
| Algorithm | Any algorithm in the IANA HTTP Message Signatures registry; shared HMAC secrets prohibited | Test vectors use Ed25519 and RSASSA-PSS with SHA-512; Ed25519 is the common choice |
Agent builder rules: sign every request at send time rather than reusing precomputed signatures (the draft says static signatures must not be treated as long-lived credentials); keep the private key in a KMS, HSM or signing service, never on the model's reach or in a container image; use one key per agent role or product, not per human, because the draft's privacy section notes keys are publicly discoverable and should represent an automation identity; and send over TLS only.
Publishing the key directory
- Location and type: serve a JWK Set at
https://<operator-domain>/.well-known/http-message-signatures-directorywith media typeapplication/http-message-signatures-directory+json. - No redirects: verifiers treat only an HTTP 200 as success and do not follow redirects, so serve it directly.
- Sign the directory: the draft's Appendix B allows the directory response to carry its own HTTP signature, proving the key holder published the keys under that authority. Cloudflare requires a signed directory and uses the tag
http-message-signatures-directoryfor it. - Caching: set HTTP cache headers deliberately; verifiers follow normal HTTP caching, so your max-age is the effective propagation time for new and removed keys.
- Rotation: publish the new key next to the old one, wait at least one cache lifetime, switch signing, then remove the old key. Removing a key is not revocation; verifiers stop accepting it only when their cached copy expires.
- Minimal exposure: list only keys in active use and avoid logging personal data from directory requests.
- Describe the agent: the individual registry draft defines a Signature Agent Card that reuses OAuth client metadata (
client_id,client_name,contacts,jwks_uri) plus aweb_bot_authobject with fields such astrigger(fetcherfor user-initiated orcrawlerfor autonomous),purpose,rate-expectation,rfc9309-product-tokenandips_uri. Publishing one helps sites write policy, but it is not yet a WG document.
Key handling for this signing key follows the same discipline as any other machine credential. See the JSON Web Key Sets (JWKS) and Signing Key Rotation Guide.
Verifying signed traffic
- Parse: read
Signature-Input,SignatureandSignature-Agent. The draft allows a 400 for unparseable signature headers. - Select the signature: use the one tagged
web-bot-auth; ignore others. - Check freshness: reject expired signatures and those with
createdin the future beyond a small skew; apply a local maximum window regardless ofexpires. - Find the key: look up the key by the pair (Signature-Agent URL, keyid), never by key alone, so one operator cannot claim another's key. If unknown, fetch from the Signature-Agent URL under strict limits.
- Verify coverage: confirm the signature covers
@authorityor@target-uriandsignature-agent, and that the authority matches your host. - Verify the signature with the algorithm bound to the key.
- Classify: the draft defines three outcomes, verified, invalid and unverified. An invalid signature claiming a known operator is an impersonation signal (T1036 Masquerading is the closest ATT&CK analogue); an unsigned request is simply unverified, and the draft says it should not be treated as evidence of anything.
- Decide: pass the outcome and operator identity to your bot and access policy; verification never replaces rate limits, account controls or business logic checks.
Hardening the verifier
| Risk | Reference | Control |
|---|---|---|
| Server-side request forgery through attacker-supplied directory URLs | CAPEC-664 Server Side Request Forgery | HTTPS only; block private, loopback and link-local ranges; no redirects; fetch from an isolated egress |
| Resource exhaustion from huge or slow directories | NIST SP 800-53 SC-5 | Cap response size, key count and fetch time; rate limit fetches per operator |
| Replay of a captured signed request | CAPEC-60 Session Replay | Short local window; nonce cache where feasible; cover method and path |
| Cache poisoning or outage-driven key loss | Draft security considerations | Discovery failures must not evict cached keys; negative cache entries no longer than five minutes |
| Proxies stripping headers | Draft security considerations | Ensure CDNs and reverse proxies pass Signature, Signature-Input and Signature-Agent to whichever layer verifies |
| Test keys in production | Draft security considerations | Block the draft's published test keys |
Verify at the edge or at the origin
| Option | Benefits | Trade-offs | Fits |
|---|---|---|---|
| CDN or WAF verification | No code; operator directory and key caching handled for you; result exposed to rules (Cloudflare populates cf.bot_management.verified_bot) | Depends on the provider's supported subset (Cloudflare does not support @query-params or @status and does not validate nonces); operator onboarding may be provider-specific | Most publishers and SaaS sites |
| Origin verification with a library | Full control over policy, coverage and replay checks; works across providers | You own SSRF hardening, caching and draft updates; the CDN must forward headers untouched | Large platforms with their own bot logic, or multi-CDN estates |
| Both | Edge filters impostors early; origin applies business policy with the verified identity | Two places to update when the draft changes | High-value targets such as ticketing, retail and travel |
Small versus large organisations: a small publisher should rely on its CDN's verification and write a handful of rules on the verified flag. A large platform or an agent operator should run its own verification tests against each major verifier, because verifiers differ on accepted header forms and supported components while the draft is young.
Where it sits among agent identity standards
- Web Bot Auth: agent to human-facing website, identifying the operator. Public key directory, no delegation.
- WIMSE HTTP signatures: workload to workload inside or across trust domains, using an RFC 9421 profile tagged
wimse-workload-to-workloadwith a Workload Identity Token. See the WIMSE for AI Agents Guide. - OAuth and MCP authorisation: agent to API with user-delegated access tokens. See the AI Agent Authorisation Guide.
- Know Your Agent programmes: business assurance about who runs an agent, which signatures can feed. See the Know Your Agent (KYA) Guide.
The webbotauth charter lists coordination with the WIMSE, OAuth, HTTPBIS, TLS and AIPREF working groups. The draft states delegation and chaining belong in a separate specification, so a signed browsing agent tells you who operates it, not which end user it serves.
Privacy trade-off: because directories are public, signing makes an agent operator identifiable on every site it visits. The individual draft draft-rescorla-anonymous-webbotauth explores anonymous bot authentication for authorisation and rate limiting, which would let sites apply limits without learning which operator is calling. It has no working group standing yet; do not design around it, but watch it if your agents serve privacy-sensitive users.
How to implement
Agent operators
- Generate an Ed25519 key per agent product in a KMS or HSM and compute its JWK thumbprint.
- Publish a signed directory at the well-known path on a domain you control; set conservative cache headers.
- Add a signing step in the HTTP client: cover
@authority,signature-agent,@methodand@path; setcreated, a shortexpires,keyid,nonceandtag. - Test against each verifier your traffic meets (for example a CDN's test endpoint) and fix header form differences.
- Register with providers that require onboarding, publish an agent card or documentation page, and keep user agent strings consistent.
- Rehearse key rotation and key compromise: overlap keys for rotation; for compromise, remove the key, reduce cache lifetimes in advance and notify providers.
Publishers and edge teams
- Turn on verification in log-only mode and record verified, invalid and unverified outcomes per operator.
- Review a few weeks of data; confirm proxies do not strip headers.
- Write policy: allow or rate limit verified operators per your content policy, challenge invalid claims to be known agents, treat unsigned traffic as before.
- Enforce gradually by path; roll back by switching rules back to log-only, never by trusting user agents again.
Detection and measurement
- Log: outcome, operator URL, keyid, covered components, signature age, directory fetch results and errors.
- Alert: spikes in invalid signatures for a known operator; one keyid used from unusual networks at volume; directory fetches to internal addresses blocked by the SSRF filter; signatures with windows far above your local maximum.
| Metric | What it shows |
|---|---|
| Share of automated traffic that is signed and verified | Adoption among the agents that matter to you |
| Invalid signature rate per claimed operator | Impersonation attempts or operator misconfiguration |
| User agents claiming a signing operator but sending no signature | Spoofing of known agents |
| Median signature age at verification | Whether operators sign per request |
| Directory fetch failures and blocked fetches | Verifier health and SSRF probing |
| (Operators) verification success rate per provider | Interoperability while the draft settles |
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev. 5 | IA-9 Service Identification and Authentication, SC-8, SC-12, SC-5, SI-4 | Authenticating automated clients, protected transport, key management, verifier resilience, monitoring |
| NIST CSF 2.0 | PR.AA-03, PR.AA-04, DE.CM-01 | Service authentication, verified identity assertions, network monitoring |
| ISO/IEC 27001:2022 Annex A | A.8.24 Use of cryptography, A.8.16 Monitoring activities | Key handling and monitoring of automated access |
| MITRE CAPEC and ATT&CK | CAPEC-664, CAPEC-60, T1036 | Verifier SSRF, replay and impersonation of known agents |
Common failure patterns
- Verified treated as allowed: a signed operator gets unlimited access, although the draft says a valid signature does not show the request is authorised.
- Long-lived or reused signatures: operators sign once and replay for hours, so any captured request can be replayed until
expires. - Keys on the agent host: the signing key sits in the container or environment where tools or injected prompts can reach it.
- Unbounded directory fetching: verifiers follow any Signature-Agent URL, including internal addresses.
- Building against the replaced draft: code and documentation still follow the expired architecture draft and break as the WG document changes.
- Headers stripped in transit: a proxy drops signature headers and all traffic silently becomes unverified.
Practitioner checklist
- Track draft-ietf-webbotauth-httpsig-protocol on datatracker and record the version you implement.
- Sign each request with a short expiry, a nonce and coverage of authority, signature-agent, method and path.
- Keep signing keys in a KMS or HSM, one per agent product, never in agent context.
- Publish a signed directory at the well-known path, without redirects, with deliberate cache headers.
- Rotate keys with overlap and plan compromise response around cache lifetimes.
- Verify with key lookup by operator URL and keyid, and treat invalid known-operator claims as impersonation.
- Harden directory fetching against SSRF and resource exhaustion.
- Run verification in log-only mode before enforcing, and keep rate limits for verified agents.
Standards and references
- IETF: HTTP Message Signatures for automated traffic (draft-ietf-webbotauth-httpsig-protocol)
- IETF: RFC 9421 HTTP Message Signatures
- IETF: Web Bot Auth working group charter
- IETF: Web Bot Auth working group documents
- IETF: Registry and Signature Agent card for Web bot auth (individual draft)
- Cloudflare: Web Bot Auth reference
- MITRE CAPEC: Server Side Request Forgery
Related NHI Mgmt Group resources: Bot vs AI Agent Traffic Management Guide · Agentic Browsers Security Guide · IETF Agent Identity and Authorisation Drafts Tracker · Bot Management Guide