Notifications
Clear all
Topic starter
01/08/2026 8:57 pm
TL;DR: McKinsey’s Lilli incident appears to have started with exposed API documentation, unauthenticated endpoints, SQL injection, and broken object-level authorization, according to CodeWall and McKinsey’s public comments. The AI-specific risk was blast-radius expansion: database writes could alter prompts, routing, and retrieval without a code deploy, which makes this a software security and governance problem before it is a model problem.
NHIMG editorial — based on content published by Promptfoo covering the McKinsey Lilli incident: public analysis of exposed API surface, SQL injection, and object-level authorization flaws
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
Questions worth separating out
Q: What breaks when AI assistants rely on ordinary backend access controls?
A: When an AI assistant depends on ordinary APIs and databases, any gap in authentication, SQL handling, or object-level authorization can change what the assistant sees or says.
Q: Why do AI assistants increase the impact of ordinary application-security flaws?
A: They increase impact because the backend data often shapes prompts, routing, retrieval, and user context.
Q: What do security teams get wrong about AI access risk?
A: Many teams focus on the model while ignoring the identity path that reaches it.
Practitioner guidance
- Audit assistant-adjacent API routes Identify every public and undocumented route that can read, write, or influence assistant state, then require authentication and authorization on each path before it reaches the database.
- Separate mutable control data from conversational data Move prompts, routing rules, retrieval settings, and access-control metadata into governed configuration stores with change approval, versioning, and rollback records.
- Test for object-level authorization at the record boundary Build tests for BOLA on every object an assistant can access, including employee records, internal knowledge objects, and client-linked data.
What's in the full analysis
Promptfoo's full article covers the operational detail this post intentionally leaves for the source:
- The reported endpoint sequence and how the researcher moved from exposed documentation to backend interaction.
- The specific SQL and object-authorization conditions described in the public writeup, including the role of JSON keys and identifiers.
- McKinsey's public response on remediation timing and its forensic statement on whether client data was accessed.
- The article's comparison of AI-security framing versus ordinary software and configuration governance.
👉 Read Promptfoo's analysis of the McKinsey Lilli AI assistant incident →
McKinsey Lilli and AI assistant exposure: what IAM teams missed?
Explore further
01/08/2026 9:05 pm
AI assistant incidents often begin as appsec failures, not model failures. The public record around McKinsey Lilli points to exposed routes, SQL injection, and broken object-level authorization before any AI-specific behaviour mattered. That means the first governance boundary is the application and the backend data plane, not the model prompt alone. Practitioners should stop treating every AI incident as a specialised AI event.
A few things that frame the scale:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
- A second signal from the same research shows that only 52% of companies can track and audit the data their AI agents access, which leaves 48% with no reliable compliance or investigation trail.
A question worth separating out:
Q: Who is accountable when an assistant leaks data through backend flaws?
A: Accountability sits across application owners, IAM teams, and data owners because the failure usually spans authentication, authorization, and data governance. If prompts and retrieval settings are mutable backend records, then the assistant’s behaviour is governed like any other application control plane. Frameworks such as OWASP and NIST CSF help assign and test that responsibility.
👉 Read our full editorial: McKinsey Lilli shows how appsec flaws reshape AI assistant behavior
