Web Scraping Defence Guide: Stop Content, Price and Data Scrapers

Anything a browser can read, a scraper can copy, so scraping defence is less about blocking a bot and more about deciding what data you expose, to whom, at what rate, and on what terms. Scrapers take product prices to undercut you, articles and images to republish, listings and profiles to resell, and, increasingly, whole sites to train or ground AI models. OWASP's automated threat catalogue names the threat OAT-011 Scraping and lists among its aliases content scraping, harvesting, comparative shopping, mirroring and, now, LLM training bots and AI RAG bots. This guide covers the scraping-specific layer: the legal and policy posture, robots.txt and AI crawler controls, API and page design that limits bulk extraction, rate limiting, detection and measurement. It is for application security, web platform and product teams. Broader bot detection and the full automated threat taxonomy are in the Bot Management Guide; this page goes deeper on scraping and does not repeat it.

Key takeaways

• robots.txt is a request, not a lock. RFC 9309 states its rules "are not a form of access authorization"; use it to state policy to compliant crawlers and enforce everything else elsewhere.
• Separate AI crawlers by purpose. Training, search and user-initiated fetchers are different agents with different tokens, and blocking one does not block the others.
• Design limits bulk extraction more than detection does. Pagination caps, authorisation on object IDs, minimal response fields and login for bulk data remove the cheap paths.
• Rate limit by identity, not only by IP. Scrapers rotate addresses; per account, per session, per API key and per object-sequence limits survive rotation.
• Scraped personal data is still personal data. Twelve privacy regulators said in 2023 that publicly available personal data remains protected and that mass scraping may be a reportable breach.

What scrapers want and how they get it

OWASP defines OAT-011 as collecting accessible data or processed output for use elsewhere, on an ongoing or periodic basis, with fake accounts, compromised accounts or no authentication at all. The target shapes the defence.

OWASP's notes on OAT-011 add a less obvious harm: scraping can reveal how an application is built and behaves, which supports reverse engineering. In ATT&CK terms, scraping public sites is reconnaissance (T1593 Search Open Websites/Domains, T1594 Search Victim-Owned Websites), which is why harvested staff directories and email formats often reappear in phishing.

Legal and policy posture

Technical controls work better when the terms behind them are clear. Decide, with legal counsel, what automated access you permit and say so in three places: terms of use, robots.txt and, where relevant, machine-readable licensing metadata.

• EU text and data mining reservation: Article 4 of Directive (EU) 2019/790 allows reproduction and extraction of lawfully accessible works for text and data mining unless the rightholder has expressly reserved those rights in an appropriate manner, such as by machine-readable means for content made publicly available online. Recital 18 lists metadata and website terms and conditions as machine-readable means. Article 3, the separate scientific research exception, is not affected by a reservation. If you want to reserve rights, make the reservation machine-readable as well as written in your terms.
• Personal data: the joint statement of 24 August 2023 by twelve data protection authorities, including the UK ICO, says organisations need a lawful basis to collect personal data even when it is publicly available, that mass scraping incidents may need to be reported as personal data breaches, and that social media companies should protect users from unlawful scraping. If your platform holds profiles, scraping defence is part of your security of processing obligations. See the AI and GDPR Guide.
• Contract and computer misuse law: whether scraping in breach of terms, or bypassing technical measures, is actionable differs by country and has been litigated repeatedly. Do not rely on a legal theory as your control; use it to support takedown and enforcement after technical measures.
• A licensed alternative: where demand is legitimate (price comparison partners, researchers, AI companies seeking a licence) an authenticated feed or API with terms turns anonymous scrapers into known partners and makes the remaining scrapers easier to spot.

robots.txt and AI crawler controls

RFC 9309, the Robots Exclusion Protocol, is an IETF Standards Track document published in September 2022. Its mechanics matter when you write rules for many crawlers.

AI crawlers by purpose

Operators now publish separate agents per purpose, and robots.txt policy should follow purpose rather than company.

Two points follow. First, a training opt-out and a search opt-out are separate decisions; many publishers allow search agents and disallow training agents. Second, user-initiated fetchers behave more like a browser on someone's behalf, so policy for them is about rate and verification, not robots.txt. Standardisation is moving: Google says it intends to support the IETF AI preferences (AIPREF) training signal in early 2027. Agent classification and signed agents are covered in the Bot vs AI Agent Traffic Management Guide and the AI Crawler and Scraper Controls guide.

• Verify before you trust a name: Google warns that user agent strings can be spoofed. Check claimed crawlers against operator-published IP lists (OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json) or reverse DNS, and treat an unverified "Googlebot" as an unknown bot.
• Enforce at the edge: a robots.txt disallow should be mirrored by an edge rule that blocks or rate limits the same verified agent if it ignores the file, and challenges unverified clients claiming to be it.

Designing pages and APIs that resist bulk extraction

Most large scraping incidents use the application exactly as designed, only faster. The OWASP API Security Top 10 2023 names the relevant weaknesses: API4 Unrestricted Resource Consumption (no limits on records per request or requests per client), API6 Unrestricted Access to Sensitive Business Flows (automated access to a flow that harms the business when used excessively), API1 Broken Object Level Authorization and API3 Broken Object Property Level Authorization (returning objects or fields the caller should not see).

• Cap page size and depth: enforce a server-side maximum on limit or page_size parameters and a maximum result depth for search, regardless of what the client asks for.
• Avoid sequential identifiers on public objects: predictable integer IDs let a scraper walk the whole table. Use non-guessable identifiers and authorise every object request.
• Return only needed fields: internal APIs often return the full record (email, phone, internal flags) while the page shows a name. Minimise the response, not just the rendering.
• Require login for bulk or sensitive data: member directories, full-resolution media, historical prices and export functions belong behind authentication with per-account quotas.
• Protect the business flow, not just the endpoint: API6 recommends identifying the flows that hurt when automated and choosing protections for them; for search and listing flows that means quotas per account and realistic human timing (OWASP ASVS 5.0 V2.4.1 and V2.4.2).
• Treat GraphQL and internal APIs as public: query depth and cost limits apply. See the GraphQL Security Guide.
• Plant canary records: unique fake listings, profiles or price points that appear only in bulk extracts let you prove where scraped data went. See the Honeytokens and Canary Credentials Guide.

Rate limiting that survives rotation

Scrapers spread requests across residential proxies and many accounts, so a single per-IP limit mostly hurts shared networks. Layer limits on several keys.

Return HTTP 429 with a Retry-After header to well-behaved clients so partners can back off, and prefer slowing or degrading responses for suspected scrapers over hard blocks, which tell them exactly when to rotate. Implementation detail is in the Rate Limiting and API Abuse Guide and the API Gateway Security Guide.

Where identity fits

Every scraper is a non-human identity, declared or not. Signed or verified crawlers, partner API keys and OAuth clients are identities you can grant, limit and revoke; anonymous scrapers are identities you have to infer. Moving legitimate automation onto issued credentials, with owners and quotas, is what lets you treat everything else as hostile. Keys embedded in mobile apps or JavaScript are routinely extracted and reused by scrapers, so scope them to the minimum and monitor their use. See the API Security Guide.

Detecting scrapers

• Coverage anomalies: a client that requests a large share of product or profile pages, in ID or alphabetical order, with no images, CSS or JavaScript loaded.
• Timing: regular inter-request intervals, activity around the clock, or bursts immediately after price or stock changes.
• Client fingerprint mismatch: a browser user agent whose TLS and HTTP behaviour does not match that browser, headless browser traits, or a claimed crawler outside its operator's published ranges.
• Account behaviour: new accounts that only search and view, never transact; many accounts sharing devices or payment details.
• Canary hits: canary records appearing on a competitor's site, a reseller listing or in an AI answer.
• Cost signals: bandwidth and compute per page view rising without matching conversions, the infrastructure symptom API4 describes.

Rollout

1. Inventory exposed data: list pages and APIs that return prices, content or personal data, and classify each by harm if bulk copied.
2. Set policy: agree permitted automation (search engines, partners, AI agents by purpose), terms and robots.txt with legal and commercial owners.
3. Fix design first: pagination caps, object authorisation, field minimisation and login for bulk data.
4. Deploy limits in monitor mode: log what each rate limit and edge rule would have done for two weeks; review false positives with support and partner teams.
5. Enforce in stages: start with the highest-harm endpoints, throttle before blocking, and keep a per-rule rollback switch.
6. Offer the licensed path: publish how partners obtain an API key or feed, then tighten anonymous access.
7. Review monthly: canary hits, top clients by object coverage and verified crawler compliance.

Small vs large organisations: a small site gets most of the value from a clear robots.txt, a CDN or WAF rate-limiting rule and page-size caps. Large marketplaces and publishers need per-account quotas, behavioural detection and often a dedicated bot management layer; see the Bot Management Buyer's Guide.

Metrics

• Share of traffic from verified agents vs unverified automation, by page class.
• Maximum object coverage by a single client per day on catalogue and profile endpoints.
• Rate-limit actions and false positive reports per week.
• Canary records detected externally and time to takedown.
• Bandwidth and compute per human page view, trended.

Framework mapping

Common failure patterns

• Relying on robots.txt to protect data. Listing /admin-export/ as disallowed advertises it; RFC 9309 is explicit that rules are not access control.
• Blocking the web page but not its API. The HTML is rate limited while the JSON endpoint behind it returns 1,000 records per call to anyone.
• Trusting the user agent. Allow-listing "Googlebot" by string lets every scraper that copies the string through.
• One opt-out for all AI. Disallowing a training token and assuming search and user-initiated agents are covered, or blocking a search agent and losing visibility in AI search by accident.
• Hard blocks that teach the scraper. Instant 403s reveal thresholds; throttling and degraded responses cost the scraper more.
• Treating public profiles as fair game. Regulators have said publicly available personal data is still protected; weak limits on member search can become a breach notification.

Practitioner checklist

• Inventory pages and APIs exposing prices, content and personal data, and rate the harm of bulk copying.
• Agree an automated access policy and reflect it in terms, robots.txt and machine-readable reservations.
• Write robots.txt groups per AI crawler purpose and verify claimed crawlers against published ranges.
• Cap page sizes, use non-sequential IDs, authorise every object and minimise response fields.
• Require authentication and per-account quotas for bulk or personal data.
• Layer rate limits on IP, session, account, API key and object coverage, starting in monitor mode.
• Plant canary records and watch for them externally.
• Offer a licensed API or feed for legitimate demand.
• Track verified vs unverified automation, object coverage, canary hits and cost per human view.

Standards and references

• IETF: RFC 9309 Robots Exclusion Protocol
• OWASP: Automated Threats to Web Applications, OAT-011 Scraping
• OWASP: API Security Top 10 2023
• OWASP: Application Security Verification Standard 5.0
• EUR-Lex: Directive (EU) 2019/790 on copyright in the Digital Single Market
• ICO: Joint statement on data scraping and data protection
• Google: Google's common crawlers
• OpenAI: Overview of OpenAI crawlers

Related NHI Mgmt Group resources: Bot Management Guide · Bot vs AI Agent Traffic Management Guide · Rate Limiting and API Abuse Guide · API Security Guide