Crawl-Delay

What Crawl-Delay Means for Crawlers and Site Operators

Crawl-Delay is a polite request, not a hard control. It tells a crawler to pause between requests so the site is less likely to experience burst load, but crawler support varies and some bots ignore it entirely. That makes it useful as a signalling mechanism, not as a reliable enforcement boundary.

Because robots.txt is publicly visible, Crawl-Delay is better understood as coordination metadata than as access control. It can shape well-behaved crawling behaviour, but it does not authenticate a bot, enforce identity, or guarantee that requests will stop at the advertised rate.

How Crawl-Delay Works in robots.txt

The directive sits alongside other robots.txt instructions and is read by crawlers that choose to honour it. The value is expressed as a number of seconds between requests, which means its practical effect depends on how the crawler interprets that number and whether it applies it per host, per user agent, or in some other implementation-specific way.

In practice, this makes Crawl-Delay a compatibility feature rather than a standardised traffic-shaping control. Search engines and other crawlers differ in whether they support it, how they parse it, and whether they treat it as a global limit or a hint tied to a specific user agent.

Why Crawl-Delay Is Used

Site operators usually use Crawl-Delay when they want to reduce request pressure from automated traffic without blocking access altogether. It is especially relevant for smaller sites, fragile back-end systems, or content estates where aggressive crawling can create unnecessary latency, log noise, or transient resource spikes.

It can also be a practical courtesy signal when a publisher prefers gradual indexing over rapid fetching. That said, if the business requirement is strict traffic governance, a robots.txt hint is usually too weak on its own and should be paired with server-side rate limiting, caching, and crawler-specific allowlisting or throttling where appropriate.

Limitations and Operational Trade-offs

Crawl-Delay is easy to publish, but easy to bypass. A crawler can ignore it, a malicious scraper can deliberately disregard it, and some legitimate crawlers may not support it at all. The result is that the directive reduces expected load only when the recipient cooperates.

The trade-off is between convenience and control. A lower-friction publishing model is simple to deploy, but it gives operators little assurance under abusive or poorly behaved crawling. If overuse becomes a concern, the real control point is usually server enforcement, not the robots.txt file itself.

Related resources from NHI Mgmt Group

• Should organisations delay production AI agents until identity governance is mature?
• When should organisations restrict or delay MCP deployment?
• When should teams delay customer-facing AI features?
• What breaks if organisations delay crypto-agility until quantum computing is mature?