Bot Management Guide: Scraping, Credential Stuffing and Scalping

Most automated attacks on web applications use features exactly as designed, a login form, a checkout, a search page, only at a speed and scale no human customer would. That is why vulnerability scanners and traditional WAF rules miss them, and why bot management has become its own discipline. This guide uses the OWASP Automated Threats to Web Applications taxonomy (OAT-001 to OAT-021) to name the problems precisely, then covers how detection works across network, TLS, browser and behavioural signals, how to defend login, checkout and content, how to let good bots and AI agents through by verifying them rather than trusting their user agent, and how to roll out and measure controls. It is written for application security, fraud and platform teams, and for anyone choosing between CDN, WAF and dedicated bot management capabilities.

Key takeaways

• Name the threat before choosing the control. Credential stuffing (OAT-008), scalping (OAT-005) and scraping (OAT-011) need different signals and different responses; a single "bot score" threshold rarely fits all three.
• Protect the business flow, not the page. Rate limits and detection should key on accounts, cards, products and sessions as well as IP addresses, because residential proxies spread attacks across thousands of clean-looking addresses.
• Challenges are a cost, not a wall. CAPTCHAs are solved by machine learning and human solving services, and a challenge token that is not validated server-side protects nothing.
• Verify good bots cryptographically or by DNS. User-agent strings are trivially spoofed; use forward-confirmed reverse DNS, published IP ranges or HTTP message signatures (the IETF Web Bot Auth work).
• Measure outcomes, not blocks. Track account takeover rate, login success ratio, inventory sold to genuine customers and infrastructure cost, not the number of requests blocked.

The OWASP automated threat taxonomy

The OWASP Automated Threats to Web Applications project defines 21 threat events, each with an identifier, so that security, fraud and product teams can describe the same attack in the same words. A new edition (v1.3) of its Automated Threat Handbook was published on 17 March 2026. The table groups the events by what they abuse; definitions are summarised from the OWASP ontology.

OWASP maps credential stuffing to CWE-799 (Improper Control of Interaction Frequency) and CWE-837 (Improper Enforcement of a Single, Unique Action). Those two weaknesses explain most of the category: the application lets one actor do something far more often than intended, or more than once when it should happen once.

How bot detection works

No single signal is reliable on its own, because each can be forged by a determined operator. Effective products and in-house systems layer several and score them together.

Challenges done properly

• Proof-of-work and non-interactive challenges: modern challenge services run small JavaScript checks instead of puzzles. Cloudflare Turnstile, for example, describes proof-of-work, proof-of-space, probing for web APIs and checks for browser quirks and human behaviour, and only shows a checkbox when risk is higher. Proof-of-work raises the cost per request; it does not stop a well-funded operator.
• Server-side validation: a challenge token must be verified by your server before the protected action. Turnstile tokens are valid for 300 seconds and can be validated once; without the server-side call, forged tokens pass.
• Bind the token to the action: check that the token was issued for the same session and flow, so a token solved on one page cannot be replayed on a login API.
• Use challenges selectively: challenge only traffic in a grey zone of risk. Challenging everyone pushes cost onto customers and accessibility, and teaches attackers your thresholds.

Defending login and account flows

Credential stuffing (OAT-008, ATT&CK T1110.004) tests username and password pairs stolen from other sites; it involves no guessing, so password complexity rules do not help. Credential cracking (OAT-007) and password spraying (T1110.003) do guess. The 23andMe breach showed the impact: stuffing against about 18,000 accounts led, through a data-sharing feature, to genetic data on nearly 7 million people.

• Remove the payoff with MFA: require MFA, ideally phishing-resistant, for customer accounts and especially for externally exposed staff applications (CIS Safeguard 6.3). Stuffed passwords then yield nothing. The Snowflake breach shows what happens when valid stolen credentials meet accounts without MFA. See the Passwordless and Passkeys Guide.
• Block known-breached passwords: NIST SP 800-63B-4 requires verifiers to compare new passwords against a blocklist of commonly used, expected or compromised values. The Pwned Passwords range API supports this without sending the password: the client sends the first five characters of the SHA-1 or NTLM hash and checks the returned suffixes locally (k-anonymity). Check at login too, and force a reset when a match appears.
• Rate limit per account and per device, not only per IP: SP 800-63B-4 requires limiting consecutive failed attempts on an authenticator to no more than 100, and allows bot challenges, increasing waits (its example is 30 seconds up to an hour) and risk-based techniques before that point. Distributed stuffing tries each account only once or twice, so watch distinct usernames per device fingerprint and per ASN as well.
• Uniform responses: return the same message and timing for unknown user and wrong password, so the login cannot be used to enumerate valid accounts (and do the same on password reset and sign-up).
• Protect every door: mobile app login APIs, legacy endpoints and partner APIs are often unprotected while the web form is guarded. Bots go to the weakest one.
• Account creation: limit sign-ups per device, payment instrument and phone number, delay privileges for new accounts, and watch for templated usernames. The GemStuffer campaign used an AI agent swarm to mass-create accounts on a package registry.
• SMS cost abuse: one-time passcode and "send me a link" endpoints are targets for cost-inflation fraud (OAT-003). Cap messages per number, per destination country and per session.

Scalping, inventory and payment abuse

• Scalping and sniping (OAT-005, OAT-013): for ticket on-sales and limited releases such as sneaker drops use a virtual queue that admits users at a controlled rate with signed, single-use queue tokens; set purchase limits per verified account, payment instrument and delivery address, not per session; and consider raffles or pre-registration for the highest-demand drops, which remove the speed advantage entirely. In the US, the Better Online Ticket Sales (BOTS) Act of 2016 prohibits circumventing a ticket issuer's security measures, access controls or other technological controls, and reselling tickets obtained that way.
• Denial of inventory (OAT-021): shorten basket hold times for scarce stock, release held stock automatically, and limit concurrent holds per account.
• Gift cards and vouchers (OAT-002): long, random codes with a separate PIN, rate-limited balance checks keyed on card number as well as client, and alerts when one client queries many distinct cards.
• Carding and card cracking (OAT-001, OAT-010): watch authorisation attempts per card, per account and per device, small-value authorisations followed by declines, and many cards against one account. Coordinate with your payment provider, whose network view sees testing across merchants.

Good bots, AI crawlers and scraping

Search engines, uptime monitors, partner integrations and, increasingly, AI agents acting for customers are automation you want. The goal is to identify them reliably and treat them according to policy, which means not trusting a user-agent string.

• AI crawlers: separate training crawlers from search and user-initiated fetchers. OpenAI, for example, documents GPTBot (training, controlled by robots.txt), OAI-SearchBot (search) and ChatGPT-User, which fetches pages when a person asks and for which "robots.txt rules may not apply". Decide policy per class rather than per company.
• Signed agents: where your edge supports Web Bot Auth, require signatures with short expiry (Cloudflare's implementation uses the web-bot-auth tag, a key ID set to the JWK thumbprint, and created and expires parameters as the main replay protection).
• Scraping (OAT-011): robots.txt does not stop a scraper that ignores it. Rate limit expensive endpoints per session and per account (CWE-799), require login for bulk data, avoid exposing complete datasets through predictable IDs or unpaginated APIs, and offer a licensed API or feed where demand is legitimate, so you can tell partners from scrapers.

Choosing responses and an approach

Blocking is only one option, and often not the best one, because it tells the operator immediately that they were detected.

• Edge features versus dedicated bot management: CDN and WAF rate limiting, IP reputation and basic challenges handle volumetric scraping and noisy stuffing, and suit smaller organisations. Retailers, ticketing, travel and financial services, where bots monetise each success, usually need dedicated capabilities with client-side signals, account-level analytics and a fraud team using the output.
• In-house detection: worth building for business-logic signals only your application can see (per-account velocity, sell-through, card counts). Leave TLS fingerprinting and browser telemetry to specialist tooling unless you have the team to maintain it as browsers change.
• Small organisations: MFA, a breached-password check, per-account rate limits and a managed challenge on login and sign-up remove most risk at low cost.

Where identity fits

• Bots target identities: credential stuffing, cracking and account creation all end in an account. Bot controls and identity controls should share signals: a login flagged as automated should raise the risk score your identity provider uses for step-up. See the Customer IAM (CIAM) Guide.
• APIs and keys: bots move to APIs when the web front end is protected. Issue API keys per partner, scope and rate limit them, and treat an unexpected surge from one key as a compromise. See the API Key Management Guide.
• AI agents are the next good bot problem: agents booking, buying and browsing for customers need an identity you can verify and authorise, not a bypass of your bot defences. Web Bot Auth signatures are one building block. See the Agentic Commerce Identity Guide and the Browser and Computer-Use Agent Guide.
• Agents as attackers: the Taiwan autonomous AI agent attack reportedly used AI agents to compromise 85 government accounts and pivot through SSO, the same credential attack at machine speed.

Rolling out and measuring bot management

1. Inventory flows: list login, sign-up, password reset, OTP, checkout, gift card, search and high-value content endpoints across web, mobile and API, and assign each a likely OAT event and an owner.
2. Baseline in monitor mode: deploy detection in log-only mode for two to four weeks; record traffic share by bot score, login success ratio, sign-ups per day and sell-through for limited products.
3. Allowlist good automation: verify search crawlers, monitors and partners by DNS, IP ranges or signatures before enforcing anything, so the first enforcement does not block them.
4. Enforce on one flow: start with login, apply challenges to the riskiest band, and compare account takeover reports and conversion against the baseline.
5. Extend flow by flow: sign-up, checkout and inventory, then content. Keep a per-rule switch so any rule can return to log-only within minutes; that is your rollback.
6. Review continuously: operators adapt within days. Review false positives with customer support weekly and retune thresholds after each major campaign or product launch.

What to log: per request, the bot score and the signals behind it, JA4 fingerprint, ASN, challenge issued and outcome, and the account, session and business object involved. Keep login and checkout events in your SIEM (CIS Safeguard 13.10 covers application layer filtering, 13.11 tuning alert thresholds). Metrics that show it is working:

• Account takeover rate: confirmed compromised accounts per 100,000 active accounts per month.
• Login success ratio: successful logins divided by attempts; a sudden drop signals stuffing even when volume looks normal.
• Challenge pass rate and false positives: share of challenged sessions that pass, and support contacts attributed to bot controls.
• Genuine sell-through: share of limited inventory bought by accounts that pass post-sale review.
• Automated traffic cost: share of origin requests and compute attributed to unwanted automation.

Common failure patterns

• Protecting the web form but not the API: the mobile login API or a legacy endpoint has no bot controls, and attackers move there within hours.
• Per-IP limits only: residential proxies give every attempt a fresh address, so the limit never triggers while thousands of accounts are tested.
• Client-side only challenges: the widget renders but the server never validates the token, so any string is accepted.
• Allowlisting by user agent: "Googlebot" in the header bypasses rate limits, and scrapers learn to claim it.
• Lockouts as the defence: locking accounts after failures turns stuffing into denial of service against real customers (OAT-015) while distributed attempts stay under the threshold.
• Nobody owns the outcome: security deploys rules, fraud sees losses, product sees conversion; without a shared metric each team tunes against the others.

Framework mapping

In NIST CSF 2.0 terms, bot management sits in PR.AA (identity management, authentication and access control) for login flows, PR.PS (platform security) for application controls, and DE.CM and DE.AE for monitoring and analysis. ISO/IEC 27001:2022 Annex A controls A.8.16 (monitoring activities) and A.8.26 (application security requirements) apply.

Practitioner checklist

• Map every exposed flow (web, mobile, API) to the OAT events it attracts and give each an owner.
• Require MFA on customer and staff accounts, and block breached passwords at registration, change and login.
• Rate limit on account, device, payment instrument and product as well as IP address.
• Validate every challenge token server-side, bound to the session and action.
• Verify good bots by reverse DNS, published ranges or Web Bot Auth signatures, never by user agent.
• Protect inventory with queues, per-identity purchase limits and short basket holds.
• Run new rules in log-only mode first and keep a per-rule switch for rollback.
• Report account takeover rate, login success ratio and genuine sell-through monthly to security, fraud and product together.

Standards and references

• OWASP: Automated Threats to Web Applications
• NIST: SP 800-63B-4, Authentication and Authenticator Management
• MITRE ATT&CK: T1110.004 Brute Force: Credential Stuffing
• MITRE CWE: CWE-799 Improper Control of Interaction Frequency
• IETF: HTTP Message Signatures for automated traffic (Web Bot Auth working group draft)
• IETF: RFC 9309, Robots Exclusion Protocol
• W3C: WebDriver (navigator.webdriver)
• Google Search Central: Verifying Googlebot and other Google crawlers

Related NHI Mgmt Group resources: Identity Fraud Prevention Guide · Customer IAM (CIAM) Guide · Password Security Guide · Agentic Commerce Identity Guide · API Key Management Guide