Notifications
Clear all
Topic starter
08/06/2026 4:49 pm
TL;DR: As AI crawlers and answer engines become a primary discovery layer, sites need machine-readable structure, clear robots.txt policy, semantic markup, and crawl monitoring to stay useful without exposing login surfaces or enabling abuse, according to WorkOS. The underlying governance problem is that visibility and permissioning for bots now sit alongside human identity controls, not outside them.
NHIMG editorial — based on content published by WorkOS: How to make your site LLM-friendly without inviting abuse
By the numbers:
- 90% of IT leaders say properly managing NHIs is essential for a successful zero-trust implementation.
- 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools.
Questions worth separating out
Q: How should security teams control bots that crawl public content without exposing login forms?
A: Security teams should split public content from identity-bearing workflows, then enforce different controls for each.
Q: Why do LLM crawlers change the identity risk model for websites?
A: LLM crawlers change the model because they turn content into a machine-facing surface that can be indexed, summarised, and replayed at scale.
Q: What breaks when robots.txt is treated like a security control?
A: What breaks is the assumption that declared policy equals enforcement.
Practitioner guidance
- Separate crawlable content from identity surfaces Keep public summaries, documentation, and blog content on distinct paths from login, signup, password reset, and admin workflows.
- Use robots.txt as policy, not protection Publish crawler guidance in robots.txt, then verify adherence through logs, CDN analytics, and behaviour-based detection.
- Instrument bot behaviour at the edge Track user-agent patterns, request depth, and abnormal bursts from automated clients.
What's in the full article
WorkOS's full article covers the operational detail this post intentionally leaves for the source:
- Example robots.txt allow and block patterns for major AI crawlers such as GPTBot, PerplexityBot, and ClaudeBot
- Schema.org markup examples for BlogPosting, FAQPage, HowTo, and Organization on public pages
- Server-side rendering and noscript guidance for content that must remain crawlable without JavaScript
- Bot detection ideas at the CDN and log layer, including prompt honeypot testing
👉 Read WorkOS's guide to making websites LLM-friendly without bot abuse →
LLM-friendly websites and bot abuse: what should IAM teams do?
Explore further
08/06/2026 6:40 pm
LLM friendliness is becoming a machine identity problem, not just an SEO problem. Once answer engines and crawlers become the first consumer of content, the governance question shifts from ranking to authorisation. The site has to distinguish between a bot that should retrieve public content and a bot that should never reach authentication or form-handling surfaces. That is an NHI governance problem because the actor is non-human, persistent, and policy-governed at the edge of the experience.
A few things that frame the scale:
- 90% of IT leaders say properly managing NHIs is essential for a successful zero-trust implementation, according to the Ultimate Guide to NHIs.
- Only 5.7% of organisations have full visibility into their service accounts, which is why non-human traffic and machine-facing surfaces need the same scrutiny as human access paths.
A question worth separating out:
Q: How can teams balance LLM visibility with abuse prevention?
A: Teams should publish structured, public content for discovery, while keeping authentication and operational endpoints outside that same machine-readable plane. Then they should monitor crawler depth, user-agent patterns, and spikes in traffic to detect misuse. Balance comes from scoping what is readable, not from trying to hide everything.
👉 Read our full editorial: LLM-friendly websites need machine readability without bot abuse
