Join our Newsletter — 33% off our NHI Course
Home› Guides› Web Bot Auth and HTTP Message Signatures for…
Guide Agent Identity & Access

Web Bot Auth and HTTP Message Signatures for Agents: Signing, Key Directories and Verification

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 27 October 2026 13 min read
On this page

Websites cannot tell a real AI agent or crawler from an impostor using the same user agent string, and IP allow lists break as soon as agents run from shared cloud ranges. Web Bot Auth answers this with cryptography: the agent signs each HTTP request with a private key under RFC 9421 HTTP Message Signatures, publishes the public keys in a well-known directory, and the site verifies the signature before deciding how to treat the request. The IETF webbotauth working group adopted its protocol document in September 2026, replacing the earlier individual architecture draft, and CDNs already verify signed traffic. This guide gives the implementation detail for both sides: what an agent builder must sign and publish, how a publisher or edge team should verify, cache and score, and which parts are still moving. It is for agent platform engineers, edge and bot management teams, and identity architects. Statuses were checked on the IETF datatracker on 27 October 2026. For classifying agent traffic and building access policy, see the Bot vs AI Agent Traffic Management Guide; this page does not repeat it.

Key takeaways

  • The current document is draft-ietf-webbotauth-httpsig-protocol-00. It replaces draft-meunier-webbotauth-httpsig-protocol, which itself replaced the expired draft-meunier-web-bot-auth-architecture. Cite and build against the WG draft and expect changes.
  • Three moving parts: a Signature-Agent header pointing at the operator's key source, an RFC 9421 signature tagged web-bot-auth covering at least the authority and that header, and a JWKS directory at /.well-known/http-message-signatures-directory.
  • Signatures prove the operator, not the permission. The draft states that a valid signature does not show the operator is honest or the request authorised; authorisation, delegation and revocation are out of scope.
  • Verifiers fetch URLs supplied by strangers. Directory discovery is an SSRF and resource exhaustion surface; the draft asks verifiers to limit size, key count, time and redirects and to block private address ranges.
  • Short signatures, overlapping keys. Generate a fresh signature per request with a short expires, rotate keys by publishing the new key alongside the old, and remember that removing a key is not instant revocation.

Document status and lineage

DocumentVersion and dateStatusRole
RFC 9421 HTTP Message SignaturesFebruary 2024Proposed StandardThe signing mechanism everything builds on
draft-ietf-webbotauth-httpsig-protocol-00, 1 September 2026WG document; header says Standards TrackSignature-Agent header, key directory format and well-known URI, verification rules
draft-meunier-web-bot-auth-architecture-05, 2 March 2026ReplacedOriginal architecture; do not cite for current behaviour
draft-meunier-http-message-signatures-directory-05, 2 March 2026Replaced by draft-meunier-webbotauth-httpsig-directoryOriginal directory specification
draft-meunier-webbotauth-registry-03, 26 June 2026Individual draftSignature Agent Cards and registries of agents
draft-rescorla-anonymous-webbotauth-01, 19 July 2026Individual draftAnonymous bot authentication for authorisation and rate limiting
draft-illyes-webbotauth-cbcp, draft-illyes-webbotauth-jafar, draft-kelley-webbotauth-sail-01, -01, -00, 22 October 2026Individual draftsCrawler best practice, JSON format for bot IP ranges, source address identity lists

The working group charter covers search crawlers, archivers, link checkers, AI training crawlers and AI agents acting for end users against sites that mainly serve humans. HTTP APIs, agent-to-agent interfaces, end-user authentication and bot reputation are out of scope. The charter milestones targeted April 2026 for sending the authentication specification to the IESG; the protocol document is still at -00, so plan for further revisions.

What the agent sends

ElementRequirement in the WG draftNotes for implementers
Signature-Agent headerDictionary structured field whose values are https URIs; optional type parameter: directory (default), jwks_uri or cimdPoints the verifier at the operator's keys. Cloudflare's current documentation shows a quoted string form; check what each verifier accepts while formats converge.
Covered componentsMust cover @authority or @target-uri, plus the signature-agent headerCovering @method, @path or @query-param is recommended to narrow what a captured signature can be reused for
created and expiresRequired; the draft recommends expires no more than 24 hours after creationUse far shorter windows in practice. Cloudflare recommends about a minute because it does not check nonces.
keyidRequired; base64url JWK SHA-256 thumbprint of the keyLets verifiers find the key in the directory
tagRequired; value web-bot-authVerifiers may ignore signatures with other tags
nonceFollows RFC 9421 replay guidanceRecommended; random per request
AlgorithmAny algorithm in the IANA HTTP Message Signatures registry; shared HMAC secrets prohibitedTest vectors use Ed25519 and RSASSA-PSS with SHA-512; Ed25519 is the common choice

Agent builder rules: sign every request at send time rather than reusing precomputed signatures (the draft says static signatures must not be treated as long-lived credentials); keep the private key in a KMS, HSM or signing service, never on the model's reach or in a container image; use one key per agent role or product, not per human, because the draft's privacy section notes keys are publicly discoverable and should represent an automation identity; and send over TLS only.

Publishing the key directory

  • Location and type: serve a JWK Set at https://<operator-domain>/.well-known/http-message-signatures-directory with media type application/http-message-signatures-directory+json.
  • No redirects: verifiers treat only an HTTP 200 as success and do not follow redirects, so serve it directly.
  • Sign the directory: the draft's Appendix B allows the directory response to carry its own HTTP signature, proving the key holder published the keys under that authority. Cloudflare requires a signed directory and uses the tag http-message-signatures-directory for it.
  • Caching: set HTTP cache headers deliberately; verifiers follow normal HTTP caching, so your max-age is the effective propagation time for new and removed keys.
  • Rotation: publish the new key next to the old one, wait at least one cache lifetime, switch signing, then remove the old key. Removing a key is not revocation; verifiers stop accepting it only when their cached copy expires.
  • Minimal exposure: list only keys in active use and avoid logging personal data from directory requests.
  • Describe the agent: the individual registry draft defines a Signature Agent Card that reuses OAuth client metadata (client_id, client_name, contacts, jwks_uri) plus a web_bot_auth object with fields such as trigger (fetcher for user-initiated or crawler for autonomous), purpose, rate-expectation, rfc9309-product-token and ips_uri. Publishing one helps sites write policy, but it is not yet a WG document.

Key handling for this signing key follows the same discipline as any other machine credential. See the JSON Web Key Sets (JWKS) and Signing Key Rotation Guide.

Verifying signed traffic

  1. Parse: read Signature-Input, Signature and Signature-Agent. The draft allows a 400 for unparseable signature headers.
  2. Select the signature: use the one tagged web-bot-auth; ignore others.
  3. Check freshness: reject expired signatures and those with created in the future beyond a small skew; apply a local maximum window regardless of expires.
  4. Find the key: look up the key by the pair (Signature-Agent URL, keyid), never by key alone, so one operator cannot claim another's key. If unknown, fetch from the Signature-Agent URL under strict limits.
  5. Verify coverage: confirm the signature covers @authority or @target-uri and signature-agent, and that the authority matches your host.
  6. Verify the signature with the algorithm bound to the key.
  7. Classify: the draft defines three outcomes, verified, invalid and unverified. An invalid signature claiming a known operator is an impersonation signal (T1036 Masquerading is the closest ATT&CK analogue); an unsigned request is simply unverified, and the draft says it should not be treated as evidence of anything.
  8. Decide: pass the outcome and operator identity to your bot and access policy; verification never replaces rate limits, account controls or business logic checks.

Hardening the verifier

RiskReferenceControl
Server-side request forgery through attacker-supplied directory URLsCAPEC-664 Server Side Request ForgeryHTTPS only; block private, loopback and link-local ranges; no redirects; fetch from an isolated egress
Resource exhaustion from huge or slow directoriesNIST SP 800-53 SC-5Cap response size, key count and fetch time; rate limit fetches per operator
Replay of a captured signed requestCAPEC-60 Session ReplayShort local window; nonce cache where feasible; cover method and path
Cache poisoning or outage-driven key lossDraft security considerationsDiscovery failures must not evict cached keys; negative cache entries no longer than five minutes
Proxies stripping headersDraft security considerationsEnsure CDNs and reverse proxies pass Signature, Signature-Input and Signature-Agent to whichever layer verifies
Test keys in productionDraft security considerationsBlock the draft's published test keys

Verify at the edge or at the origin

OptionBenefitsTrade-offsFits
CDN or WAF verificationNo code; operator directory and key caching handled for you; result exposed to rules (Cloudflare populates cf.bot_management.verified_bot)Depends on the provider's supported subset (Cloudflare does not support @query-params or @status and does not validate nonces); operator onboarding may be provider-specificMost publishers and SaaS sites
Origin verification with a libraryFull control over policy, coverage and replay checks; works across providersYou own SSRF hardening, caching and draft updates; the CDN must forward headers untouchedLarge platforms with their own bot logic, or multi-CDN estates
BothEdge filters impostors early; origin applies business policy with the verified identityTwo places to update when the draft changesHigh-value targets such as ticketing, retail and travel

Small versus large organisations: a small publisher should rely on its CDN's verification and write a handful of rules on the verified flag. A large platform or an agent operator should run its own verification tests against each major verifier, because verifiers differ on accepted header forms and supported components while the draft is young.

Where it sits among agent identity standards

  • Web Bot Auth: agent to human-facing website, identifying the operator. Public key directory, no delegation.
  • WIMSE HTTP signatures: workload to workload inside or across trust domains, using an RFC 9421 profile tagged wimse-workload-to-workload with a Workload Identity Token. See the WIMSE for AI Agents Guide.
  • OAuth and MCP authorisation: agent to API with user-delegated access tokens. See the AI Agent Authorisation Guide.
  • Know Your Agent programmes: business assurance about who runs an agent, which signatures can feed. See the Know Your Agent (KYA) Guide.

The webbotauth charter lists coordination with the WIMSE, OAuth, HTTPBIS, TLS and AIPREF working groups. The draft states delegation and chaining belong in a separate specification, so a signed browsing agent tells you who operates it, not which end user it serves.

Privacy trade-off: because directories are public, signing makes an agent operator identifiable on every site it visits. The individual draft draft-rescorla-anonymous-webbotauth explores anonymous bot authentication for authorisation and rate limiting, which would let sites apply limits without learning which operator is calling. It has no working group standing yet; do not design around it, but watch it if your agents serve privacy-sensitive users.

How to implement

Agent operators

  1. Generate an Ed25519 key per agent product in a KMS or HSM and compute its JWK thumbprint.
  2. Publish a signed directory at the well-known path on a domain you control; set conservative cache headers.
  3. Add a signing step in the HTTP client: cover @authority, signature-agent, @method and @path; set created, a short expires, keyid, nonce and tag.
  4. Test against each verifier your traffic meets (for example a CDN's test endpoint) and fix header form differences.
  5. Register with providers that require onboarding, publish an agent card or documentation page, and keep user agent strings consistent.
  6. Rehearse key rotation and key compromise: overlap keys for rotation; for compromise, remove the key, reduce cache lifetimes in advance and notify providers.

Publishers and edge teams

  1. Turn on verification in log-only mode and record verified, invalid and unverified outcomes per operator.
  2. Review a few weeks of data; confirm proxies do not strip headers.
  3. Write policy: allow or rate limit verified operators per your content policy, challenge invalid claims to be known agents, treat unsigned traffic as before.
  4. Enforce gradually by path; roll back by switching rules back to log-only, never by trusting user agents again.

Detection and measurement

  • Log: outcome, operator URL, keyid, covered components, signature age, directory fetch results and errors.
  • Alert: spikes in invalid signatures for a known operator; one keyid used from unusual networks at volume; directory fetches to internal addresses blocked by the SSRF filter; signatures with windows far above your local maximum.
MetricWhat it shows
Share of automated traffic that is signed and verifiedAdoption among the agents that matter to you
Invalid signature rate per claimed operatorImpersonation attempts or operator misconfiguration
User agents claiming a signing operator but sending no signatureSpoofing of known agents
Median signature age at verificationWhether operators sign per request
Directory fetch failures and blocked fetchesVerifier health and SSRF probing
(Operators) verification success rate per providerInteroperability while the draft settles

Framework mapping

FrameworkReferenceRelevance
NIST SP 800-53 Rev. 5IA-9 Service Identification and Authentication, SC-8, SC-12, SC-5, SI-4Authenticating automated clients, protected transport, key management, verifier resilience, monitoring
NIST CSF 2.0PR.AA-03, PR.AA-04, DE.CM-01Service authentication, verified identity assertions, network monitoring
ISO/IEC 27001:2022 Annex AA.8.24 Use of cryptography, A.8.16 Monitoring activitiesKey handling and monitoring of automated access
MITRE CAPEC and ATT&CKCAPEC-664, CAPEC-60, T1036Verifier SSRF, replay and impersonation of known agents

Common failure patterns

  • Verified treated as allowed: a signed operator gets unlimited access, although the draft says a valid signature does not show the request is authorised.
  • Long-lived or reused signatures: operators sign once and replay for hours, so any captured request can be replayed until expires.
  • Keys on the agent host: the signing key sits in the container or environment where tools or injected prompts can reach it.
  • Unbounded directory fetching: verifiers follow any Signature-Agent URL, including internal addresses.
  • Building against the replaced draft: code and documentation still follow the expired architecture draft and break as the WG document changes.
  • Headers stripped in transit: a proxy drops signature headers and all traffic silently becomes unverified.

Practitioner checklist

  • Track draft-ietf-webbotauth-httpsig-protocol on datatracker and record the version you implement.
  • Sign each request with a short expiry, a nonce and coverage of authority, signature-agent, method and path.
  • Keep signing keys in a KMS or HSM, one per agent product, never in agent context.
  • Publish a signed directory at the well-known path, without redirects, with deliberate cache headers.
  • Rotate keys with overlap and plan compromise response around cache lifetimes.
  • Verify with key lookup by operator URL and keyid, and treat invalid known-operator claims as impersonation.
  • Harden directory fetching against SSRF and resource exhaustion.
  • Run verification in log-only mode before enforcing, and keep rate limits for verified agents.

Standards and references

Related NHI Mgmt Group resources: Bot vs AI Agent Traffic Management Guide · Agentic Browsers Security Guide · IETF Agent Identity and Authorisation Drafts Tracker · Bot Management Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 27 October 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.