Notifications
Clear all
Topic starter
08/06/2026 4:46 pm
TL;DR: Web teams are shifting from blocking all bots to selectively welcoming LLMs and AI agents, while still defending against scraping, credential abuse, and fake signups, according to WorkOS. That reversal turns content discovery, authentication, and abuse detection into an identity governance problem, not just a web UX problem.
NHIMG editorial — based on content published by WorkOS: From blocking bots to optimizing for LLMs: How the web flipped its script
Questions worth separating out
Q: How should security teams govern LLM access to public content and APIs?
A: Treat LLM access as a separate machine identity problem, not a simple extension of web publishing.
Q: Why do AI-friendly websites still need bot and fraud controls?
A: Because helpful automation and hostile automation use similar patterns at the edge.
Q: What do security teams get wrong about robots.txt and allowlists?
A: They often treat allowlisting as proof of trust.
Practitioner guidance
- Define separate policy for machine-readable surfaces Classify public docs, AI-specific paths, and transactional APIs separately so allowlisting for crawlers does not imply access to operational systems.
- Review structured metadata as an exposure surface Audit schema.org, OpenAPI, GraphQL introspection, and LLM summary fields for data that helps discovery but reveals more than intended.
- Pair allowlists with abuse telemetry Use login, signup, and request-behaviour signals to distinguish trusted machines from hostile automation, especially at the edge and auth layer.
What's in the full article
WorkOS's full article covers the operational detail this post intentionally leaves for the source:
- Specific examples of robots.txt allowlisting and AI-specific crawl paths that teams can adapt.
- Concrete documentation and metadata patterns for making products easier for LLMs to understand.
- Operational examples of selective openness alongside abuse detection at the edge.
- Details on how WorkOS positions Radar for login abuse and suspicious automation visibility.
👉 Read WorkOS's analysis of how websites are adapting for LLMs and AI agents →
LLM-ready content and machine access: what IAM teams should watch?
Explore further
08/06/2026 6:34 pm
LLM readiness is now an identity governance problem, not a web-only concern. Once content, docs, and APIs are tuned for machine consumption, the question becomes which non-human actors are authorised to interpret, traverse, or act on them. That is a governance boundary, not a UX tweak. Practitioners should treat machine legibility as part of the access model.
A few things that frame the scale:
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
- 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools.
A question worth separating out:
Q: How can organisations balance AI discovery with least privilege?
A: By limiting machine-readable exposure to the minimum content needed for discovery, while keeping operational systems, secrets, and sensitive workflows out of reach. Least privilege for machines means the model can understand what is public, but cannot infer or execute what is not intended for automated use.
👉 Read our full editorial: LLM-ready web design is changing how companies govern machine access
