Topic starter
01/08/2026 8:57 pm
TL;DR: McKinsey's Lilli incident appears to have started with exposed API documentation, unauthenticated endpoints, SQL injection, and cross-user access, according to Promptfoo, showing how ordinary application flaws can alter AI assistant behaviour without a model jailbreak. When prompts, routing rules, and retrieval settings are mutable application data, database access becomes a governance problem before it becomes an AI problem.
Editorial analysis by NHI Mgmt Group, based on content published by Promptfoo: “McKinsey's Lilli Looks More Like an API Security Failure Than a Model Jailbreak”.
By the numbers:
- 72 percent of the firm was active on the platform.
Key questions
Q: What breaks when an AI assistant's backend data is writable by ordinary application access?
A: The assistant stops being a controlled interface and starts reflecting whoever can alter its prompts, routing rules, retrieval metadata, or user context.
Q: Why do exposed API routes and unauthenticated endpoints matter so much for AI assistants?
A: Because they often sit in front of the real control plane for assistant state.
Q: How should teams test broken object-level authorization in AI assistant workflows?
A: They should test every object the assistant can access, not just the login path.
Practitioner guidance
- Audit assistant-facing endpoints Map every route that can read or write assistant state, including undocumented and internal-only paths, and require authentication on all of them.
- Treat prompt and retrieval state as privileged data Classify prompts, routing rules, retrieval metadata, and user-history tables as security-sensitive objects with change control and access review.
- Test for BOLA across assistant workflows Verify that every object the assistant can return is checked against the caller's access rights, including conversation history and linked knowledge records.
Bottom line: The Lilli incident shows how exposed API surfaces, unsafe SQL paths, and broken object-level authorization can alter AI assistant behaviour without compromising the model directly.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
01/08/2026 9:05 pm
Most enterprise AI incidents will still be software incidents first: The public record on Lilli shows that exposed endpoints, SQL injection, and broken object-level authorization can shape assistant behaviour without touching the model weights. That means the governing problem is often the application and identity layer surrounding the model, not the model itself. Teams that separate AI governance from application security are missing the real attack surface.
A few things that frame the scale:
- 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
- Secrets management is a top five cybersecurity priority for only 33% of organisations, behind cloud security (45%), API security (42%), and endpoint security (36%), according to the 2024 State of Secrets Management Survey.
A question worth separating out:
Q: When does SQL injection become an AI governance problem instead of just an AppSec issue?
A: It becomes an AI governance problem when the database stores instructions, routing logic, retrieval metadata, or context that determines assistant behaviour. At that point, a successful write can alter what the model retrieves or outputs. Security teams should treat those database paths as behaviour-control surfaces, not as ordinary data stores.
👉 Read our full editorial: McKinsey Lilli shows how appsec flaws reshape AI assistant behavior