A bot management platform sits in the request path of your most valuable pages, so a rushed rollout either blocks real customers and partners or ships in a monitoring mode that attackers never notice. This playbook covers rolling out bot management as a product class: choosing which flows to protect first, picking an integration point, building the configuration and administration securely, running in observe mode, moving to enforcement in stages, and keeping a tested rollback. It is vendor-neutral and based on the OWASP Automated Threats to Web Applications handbook, the OWASP Credential Stuffing Prevention Cheat Sheet, NIST SP 800-53, the Robots Exclusion Protocol (RFC 9309) and published crawler verification methods. It is written for web and application security engineers, platform and edge teams, and fraud teams. How bot detection works is covered in the Bot Management Guide; product selection is in the Bot Management Buyer's Guide.
Key takeaways
- Protect named flows, not "the site". Start with the endpoints where automation costs money or accounts: login, registration, password reset, checkout, search and pricing APIs.
- Observe before you block. Run every new policy in log-only mode, measure what it would have blocked, and review the false positives with the business before enforcing.
- Mobile apps and APIs need their own plan. Browser signals do not exist for native apps and partner APIs; plan SDKs, attestation or API authentication for those channels.
- Good bots must be verified, not trusted by user agent. Allow search crawlers, monitoring and partners by verified identity, and decide your stance on AI crawlers and agents explicitly.
- The policy console is a control plane for your front door. Whoever can change allow lists or switch a rule to monitor can quietly switch protection off.
Phase 0: map the automated threats and the flows
The OWASP Automated Threat Handbook gives a shared vocabulary of 21 automated threats (OAT-001 to OAT-021), defined as misuse of valid functionality by software rather than exploitation of bugs. Use it to turn "we have a bot problem" into a scoped list of flows, threats and owners.
| Flow | OWASP automated threats | Business owner | Rollout priority (NHI Mgmt Group recommendation) |
|---|---|---|---|
| Login and token refresh | OAT-008 Credential Stuffing, OAT-007 Credential Cracking | Digital identity or CIAM team | First |
| Registration | OAT-019 Account Creation, OAT-009 CAPTCHA Defeat | Growth, fraud | First |
| Checkout and payment | OAT-001 Carding, OAT-010 Card Cracking | Payments, fraud | First where cards are accepted |
| Limited inventory and booking | OAT-005 Scalping, OAT-021 Denial of Inventory, OAT-013 Sniping | Commerce | Second, or first during launches |
| Product, price and content pages, public APIs | OAT-011 Scraping, OAT-018 Footprinting | Product, legal | Second |
| Vouchers, gift cards, loyalty | OAT-002 Token Cracking, OAT-012 Cashing Out | Marketing, fraud | Second |
| Forms, reviews, messaging | OAT-017 Spamming, OAT-016 Skewing | Trust and safety | Third |
For each flow record the current loss or volume (failed logins, fake accounts, chargebacks, scraping load), because these become the baseline that proves the rollout worked. Also record the legitimate automation that uses the flow: partners, aggregators, monitoring, accessibility tools and your own test suites.
Phase 1: choose the integration point
| Integration point | How it works | Strengths | Watch for |
|---|---|---|---|
| CDN or edge | Decisions made at the provider's edge before traffic reaches origin | Absorbs volume; quick to deploy | Origin must reject traffic that bypasses the edge; decisions must reach application logs |
| Reverse proxy, WAF or API gateway module | Plug-in or connector in your own proxy layer | Fits existing routing and logging | Added latency; module updates follow your change process |
| Application middleware or server-side API call | Application asks the platform for a verdict per request or per sensitive action | Rich business context, precise per-action decisions | Fail-open or fail-closed choice when the service is slow |
| Client-side script and mobile SDK | Collects device and behaviour signals in browser or app | Signals not visible on the network | Script integrity, content security policy, privacy notices, SDK release cycle |
- Origin lockdown: with an edge deployment, restrict origin access to the edge provider (allow lists, mutual TLS or a shared secret header rotated regularly) or attackers will go around it.
- Failure mode: decide per flow whether to fail open (keep the site up, lose protection) or fail closed. Our recommendation: fail open for browsing, and add compensating rate limits on login and checkout for the failure window.
- Client signals and privacy: client-side collection is device fingerprinting, which may be subject to cookie and similar-technologies rules and data protection law in your markets. Involve privacy before enabling it and update notices. See the Consent Management Guide.
Phase 2: secure the configuration and the console
Hardening of a running platform is in Securing the bot management platform itself. Decisions to make while building:
- Roles: separate policy authors, approvers, analysts and read-only users. Changes that switch a rule from block to monitor, add allow list entries or disable protection on a path should require a second person (NIST SP 800-53 AC-5 Separation of Duties, CM-3 Configuration Change Control).
- Admin access: single sign-on through the workforce identity provider with phishing-resistant MFA; no local accounts other than a sealed break-glass account.
- Configuration as code: manage policies through the platform API or infrastructure-as-code with review and drift detection, so an emergency change made in the console is visible.
- API keys and tokens: server-side verdict calls, log exports and SDK keys are non-human credentials. Give each an owner, scope and rotation date, and keep them out of client code except for keys designed to be public. See the API Key Management Guide.
- Allow lists: every exception needs an owner, reason and expiry. Allow lists by IP or user agent are the easiest way for an attacker, or a hurried colleague, to defeat the platform.
- Script integrity: serve client scripts from a controlled origin and cover them in your content security policy and script inventory, since they run on payment and login pages.
Phase 3: observe mode and baselining
- Deploy in log-only: collect verdicts for every in-scope flow without acting on them. Run long enough to see weekly cycles, a marketing campaign and, for retailers, a busy trading period if possible.
- Join verdicts to outcomes: compare "would block" decisions with login success, fraud cases, chargebacks and fake account reports. This is the evidence the business will ask for.
- Identify good automation: list partners, monitoring, accessibility tools and internal tests that would be blocked, and decide how each will authenticate.
- Find blind spots: traffic that never reaches the platform (alternate hostnames, old API versions, mobile endpoints) is where attackers move once enforcement starts.
Phase 4: verified good bots, partners and AI agents
- Search crawlers: verify by reverse DNS then forward DNS, or against the operator's published IP range files. Google documents both methods, including checking that the reverse lookup ends in googlebot.com, google.com or googleusercontent.com and that the forward lookup returns the original IP. Never allow by user agent string alone.
- Robots.txt is advisory: RFC 9309 defines the Robots Exclusion Protocol, but it is not access control. Use it to state your policy to well-behaved crawlers and enforce with the platform.
- Signed agents: the IETF Web Bot Auth working group is developing a way for automated clients to sign requests with HTTP Message Signatures (RFC 9421), publishing keys at a
Signature-AgentURL. As checked on 11 November 2026 it is an Internet-Draft (draft-ietf-webbotauth-httpsig-protocol-01), so details may change; it identifies the signer only and does not say whether the signer is trustworthy. See Web Bot Auth and HTTP Message Signatures for Agents. - Partners and aggregators: move them to authenticated APIs with their own credentials and quotas rather than allowing them to scrape the customer interface.
- AI crawlers and agents: decide per category (training crawlers, search and answer engines, user-directed agents) whether to allow, rate-limit, charge or block, and record the decision. See the Bot vs AI Agent Traffic Management Guide.
Phase 5: staged enforcement and rollback
- Pick responses per flow: the OWASP credential stuffing cheat sheet recommends graduated responses: challenges only for suspicious requests, temporary rather than permanent IP mitigation, degradation such as delays or proof-of-work, and notifying users of unusual security events. Hard blocks suit clear-cut automation; challenges suit uncertain traffic.
- Enforce on the highest-confidence verdicts first: block known-bad automation on login and registration, challenge the uncertain band, leave everything else in observe.
- Canary: enforce on a share of traffic or one market first, watching conversion, login success and support contacts by segment.
- Accessibility: test challenges with assistive technology and provide an alternative path. See CAPTCHA and bot challenges.
- Rollback: every policy has a documented switch back to monitor, owned by on-call engineers, and the switch itself is logged and alerted so it cannot become a silent bypass.
- Handover: agree who tunes rules, who reviews allow lists and how fraud and security teams share findings. Credential stuffing response is in the Credential Stuffing Attack Response Playbook.
Decision guidance
- Edge or in-application: edge deployment gives volume absorption and speed; in-application verdicts give precise decisions on sensitive actions. Larger estates often use both, edge for volume and application calls for login, payment and account changes.
- Bot management or rate limiting alone: rate limits handle crude bursts but not distributed, low-volume attacks from residential proxies, which the OWASP cheat sheet highlights. If login or checkout abuse is material, rate limiting alone is not enough.
- Bot controls or stronger authentication: MFA and passkeys reduce the value of stuffing at the account level; bot management reduces volume and cost. They complement each other; see the Account Takeover Prevention Guide.
- Small versus large organisations: smaller sites can rely on the default bot features of their CDN or web application firewall and protect login and registration first; large retailers, ticketing and financial services need a dedicated platform, a fraud team integration and event-specific playbooks for launches.
Detection and measurement
Send verdicts, challenge outcomes, policy changes and allow list edits to the SIEM and fraud tooling. Alert on policy changes to monitor mode, new allow list entries, sudden drops in verdict volume (traffic bypassing the platform) and challenge solve rates that jump, which the OWASP cheat sheet notes can signal that a challenge has been broken.
| Metric | Why it matters |
|---|---|
| Credential stuffing attempts reaching the password check | Direct effect on account takeover risk |
| Fake accounts created per 1,000 registrations | Registration abuse outcome |
| Share of traffic to protected flows that passes through the platform | Coverage and bypass |
| False positive rate from customer complaints and challenge abandonment | Friction that drives pressure to switch protection off |
| Allow list entries past expiry | Hygiene of the main bypass mechanism |
| Origin load and infrastructure cost from automated traffic | Business benefit of scraping control |
Framework mapping
| Rollout element | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | Other references |
|---|---|---|---|
| Login abuse defences | AC-7 Unsuccessful Logon Attempts | PR.AA-03 | ATT&CK T1110.004 Credential Stuffing; OWASP OAT-008 |
| Volume and denial protection | SC-5 Denial-of-service Protection | PR.IR-04 | ATT&CK T1499 Endpoint Denial of Service; OWASP OAT-015 |
| Request inspection and filtering | SC-7 Boundary Protection, SI-10 Information Input Validation | PR.IR-01 | OWASP automated threat handbook |
| Policy change control | CM-3 Configuration Change Control, AC-5 Separation of Duties | PR.PS-01 | Configuration as code |
| Monitoring | SI-4 System Monitoring, AU-6 Audit Record Review, Analysis, and Reporting | DE.CM-01, DE.AE-02 | SIEM and fraud integration |
Common failure patterns
- Stuck in monitor mode: the platform is bought, deployed and never moved to enforcement because nobody owns the decision. Optional controls on login are how large stuffing campaigns succeed, as in the 23andMe credential stuffing breach.
- Origin left open: the edge protects the main hostname while the origin, an old API version or the mobile endpoint is reachable directly.
- User agent allow lists: "Googlebot" or a partner name in the user agent bypasses protection without verification.
- Allow lists that never expire: exceptions for a campaign, test or partner remain years later.
- Challenge everyone: blanket CAPTCHAs raise abandonment and accessibility complaints, and the business demands they be removed.
- No fraud team in the loop: verdicts are not joined to chargebacks or account takeover cases, so the platform cannot prove value or be tuned.
Practitioner checklist
- Map each protected flow to OWASP automated threats, an owner and a loss baseline.
- Choose edge, proxy, application or SDK integration per channel and lock down the origin.
- Decide fail-open or fail-closed per flow, with compensating rate limits.
- Restrict console roles, require two-person approval for monitor switches and allow lists.
- Run every policy in observe mode and join verdicts to fraud and login outcomes.
- Verify good bots by DNS or published ranges, never by user agent.
- Move partners to authenticated APIs and record your AI crawler stance.
- Enforce in stages with a canary, an accessible alternative and a logged rollback switch.
- Review metrics and expired allow lists monthly.
Standards and references
- OWASP: Automated Threats to Web Applications
- OWASP: Credential Stuffing Prevention Cheat Sheet
- IETF: RFC 9309, Robots Exclusion Protocol
- IETF: HTTP Message Signatures for automated traffic (Web Bot Auth working group draft)
- Google Search Central: Verifying Googlebot and other Google crawlers
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Credential Stuffing
Related NHI Mgmt Group resources: Bot management platform health check · Bot management platform integration security guide · Fake Account Creation and Sign-up Fraud Guide · Bot Traffic and Automated Attack Statistics Guide