McKinsey Lilli shows how appsec flaws reshape AI assistant behavior

At a glance

What this is: This is an analysis of how ordinary application-security flaws can change AI assistant behavior by giving backend access to mutable prompts, routing rules, and retrieval data.

Why it matters: It matters because identity and access teams now have to govern not just who can reach an AI assistant, but who can alter the data and permissions that shape its outputs.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).

👉 Read Promptfoo's analysis of the McKinsey Lilli AI assistant incident

Context

McKinsey Lilli, as described in the public record, is less a model-jailbreak story than an application-security incident that happened to reach an AI system. The reported chain began with exposed API documentation, unauthenticated endpoints, unsafe SQL construction, and broken object-level authorization, which are familiar control failures in any enterprise web application.

For identity teams, the lesson is sharper than the usual AI-security framing. If prompts, routing rules, retrieval settings, and user history live as mutable backend data, then application writes can alter what the assistant sees and says without changing the model itself. That moves the governance problem into IAM, application security, and configuration control.

This is a typical pattern, not an edge case. Enterprise assistants are increasingly thin layers over ordinary APIs and databases, so the identity risk often sits in the controls around the system rather than inside the model layer itself.

Key questions

Q: What breaks when AI assistants rely on ordinary backend access controls?

A: When an AI assistant depends on ordinary APIs and databases, any gap in authentication, SQL handling, or object-level authorization can change what the assistant sees or says. The model does not need to be compromised for the system to misbehave. The backend becomes the real control plane, so standard application-security failures can turn into identity and data exposure.

Q: Why do AI assistants increase the impact of ordinary application-security flaws?

A: They increase impact because the backend data often shapes prompts, routing, retrieval, and user context. A write to those records can change behaviour without a code deploy, which makes database compromise a control-plane event. That is why identity teams need to govern both access to the application and authority over the data that drives the model.

Q: What do security teams get wrong about AI access risk?

A: Many teams focus on the model while ignoring the identity path that reaches it. If a service account or token can invoke AI infrastructure, then that credential becomes the real control point. The mistake is treating AI risk as a model problem instead of an access governance problem.

Q: Who is accountable when an assistant leaks data through backend flaws?

A: Accountability sits across application owners, IAM teams, and data owners because the failure usually spans authentication, authorization, and data governance. If prompts and retrieval settings are mutable backend records, then the assistant’s behaviour is governed like any other application control plane. Frameworks such as OWASP and NIST CSF help assign and test that responsibility.

Technical breakdown

Exposed API surface and missing authentication

The first weakness in this incident class is simple exposure: public API documentation and endpoints that do not enforce authentication or standard middleware checks. When an assistant or its backend is reachable through undocumented routes, the security boundary shifts away from the intended access path. Attackers do not need model-specific tricks if they can reach the application layer directly. In practice, this is ordinary web security failing in front of an AI-shaped interface, which means the first control question is whether every route that can influence assistant behaviour is protected with the same rigor as the rest of the application estate.

Practical implication: inventory assistant-adjacent routes and verify that authentication, authorization, and logging are enforced before any backend write or read occurs.

SQL injection through dynamic identifiers

The reported SQL injection pattern matters because bind variables protect values, not schema elements such as table names, column names, or sort order. If application code concatenates attacker-controlled JSON keys, identifiers, or field names into SQL syntax, it creates a path for backend manipulation even when normal values are parameterized. That is why this class of flaw survives WAFs and superficial secure-coding checks. In AI systems, the impact is larger when the database stores prompts, routing metadata, or access policies, because a write can change system behaviour as well as data.

Practical implication: review any assistant workflow that treats request keys or JSON paths as dynamic SQL identifiers and remove that pattern from write paths.

Broken object-level authorization in assistant backends

Broken object-level authorization, often called BOLA, occurs when a system accepts an object identifier and returns or modifies a record without verifying that the caller is entitled to that object. For AI assistants, this can expose another user’s history, internal knowledge objects, or client-linked records even when the model itself is not compromised. The deeper issue is that assistants often sit on top of shared data stores, so an authorization miss in the backend becomes a conversational leak at the front end. That is a governance failure in access control, not a hallucination problem.

Practical implication: test assistant backends for object-level authorization on every object that can be read, written, or surfaced into model context.

Threat narrative

Attacker objective: The attacker’s objective is to reach assistant-shaping backend data and use it to alter, expose, or infer information through the AI interface.

1. Entry occurred through exposed API documentation and unauthenticated endpoints that allowed the attacker to interact with the application without normal access controls.
2. Escalation followed when unsafe SQL construction and missing object-level authorization let backend access reach data that shaped assistant behaviour, including prompts and retrieval context.
3. Impact was the ability to change what the AI assistant could see or return, which expands blast radius even when no model jailbreak occurs.

Breaches seen in the wild

• Meta AI Instagram Account Takeover — 20,225 Instagram accounts hijacked via compromised Meta AI support chatbot with overprivileged access.
• Replit AI Tool Database Deletion — Replit vibe coding AI assistant deletes live production database and creates 4,000 fake user records.

Read our 52 NHI Breaches Analysis report for a comprehensive view of breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI assistant incidents often begin as appsec failures, not model failures. The public record around McKinsey Lilli points to exposed routes, SQL injection, and broken object-level authorization before any AI-specific behaviour mattered. That means the first governance boundary is the application and the backend data plane, not the model prompt alone. Practitioners should stop treating every AI incident as a specialised AI event.

Mutable assistant data creates an identity blast radius. If prompts, routing rules, retrieval metadata, and user context are stored as editable rows, then database write access can change the assistant’s behaviour without a code deploy. That collapses the distinction between data access and control-plane access, because whoever can alter those records can influence the identity context the model consumes. The implication is that configuration governance now belongs in identity and access reviews, not only in application release management.

Broken object-level authorization is the hidden identity failure mode in enterprise assistants. The same BOLA gap that leaks a document or record in a conventional application can surface another user’s context through an AI interface. That is not a new class of bug, but it becomes a new class of breach when model responses normalise the leak into an apparently legitimate answer. Teams should treat object scoping as part of assistant identity design.

Credential and object scope must be governed together, not separately. An assistant that can read internal knowledge, employee records, and client-linked objects inherits the trust boundaries of every upstream system it touches. When those scopes are loosely defined, access reviews become incomplete because the review artefact says who can reach the application, not what the application can disclose after backend compromise. Practitioners need a single control view across auth, data access, and retrieval policy.

Prompt security is only one layer of AI governance. This incident shape shows that prompt hardening will not compensate for unsafe SQL, undocumented routes, or weak backend authorization. The field keeps over-indexing on the visible conversational layer, but the real failure often sits in the ordinary controls that surround it. Security teams should reframe AI governance as application and identity governance with a model on top.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
• A second signal from the same research shows that only 52% of companies can track and audit the data their AI agents access, which leaves 48% with no reliable compliance or investigation trail.
• For a deeper governance lens, see OWASP Top 10 for Agentic Applications 2026 for the risk categories most likely to surface when backend control and AI behaviour intersect.

What this signals

Mutable assistant state is now a governance surface. When prompts, retrieval policy, and routing live as editable backend records, the control problem shifts from model safety to access governance. Teams should treat those records like privileged configuration, because a database write can become a behavioural change without any code release.

The next programme gap will be evidence, not theory. If only 52% of organisations can track and audit what AI systems access, then incident response, compliance, and legal review will all suffer from the same blind spot when an assistant sits on shared backend data.

Identity blast radius: this is the point where access to an application becomes access to the system’s behaviour. That means IAM, data security, and application security need a shared control model, especially where AI assistants read internal knowledge or employee-linked objects.

For practitioners

• Audit assistant-adjacent API routes Identify every public and undocumented route that can read, write, or influence assistant state, then require authentication and authorization on each path before it reaches the database.
• Separate mutable control data from conversational data Move prompts, routing rules, retrieval settings, and access-control metadata into governed configuration stores with change approval, versioning, and rollback records.
• Test for object-level authorization at the record boundary Build tests for BOLA on every object an assistant can access, including employee records, internal knowledge objects, and client-linked data.
• Review SQL paths that use request keys as identifiers Inspect any code that concatenates JSON keys, field names, or sort parameters into SQL syntax and replace it with safe server-side mapping.
• Extend access reviews to assistant context scope Review not only who can access the assistant, but which backend objects, prompts, and retrieval policies that identity can influence or expose.

Key takeaways

• McKinsey Lilli is best understood as a backend access-control incident that reached an AI interface, not as a model jailbreak.
• The reported failure chain shows how exposed routes, SQL injection, and broken object-level authorization can change assistant behaviour without changing code.
• Identity teams need to govern who can alter prompts, retrieval data, and object scope, because those records now shape what the assistant can reveal.

Key terms

• Broken Object-Level Authorization: A failure to check whether an authenticated identity may access a specific object, record, or device. The request succeeds because the credential is valid, but the application does not enforce per-object entitlement. In NHI environments, this turns a legitimate token into cross-resource exposure.
• Mutable Assistant State: Backend data that changes what an AI assistant can see, retrieve, or say, including prompts, routing rules, retrieval settings, and user context. When this state is writable by application paths, it becomes part of the control plane and must be governed like privileged configuration, not ordinary content.
• Identity Blast Radius: The amount of damage a compromised identity can cause across systems, data, and infrastructure. In NHI environments, it is shaped by permissions, network reach, and administrative capability rather than by the credential alone. Reducing blast radius is a containment strategy that limits lateral movement and data exposure.

What's in the full analysis

Promptfoo's full article covers the operational detail this post intentionally leaves for the source:

• The reported endpoint sequence and how the researcher moved from exposed documentation to backend interaction.
• The specific SQL and object-authorization conditions described in the public writeup, including the role of JSON keys and identifiers.
• McKinsey's public response on remediation timing and its forensic statement on whether client data was accessed.
• The article's comparison of AI-security framing versus ordinary software and configuration governance.

👉 Promptfoo's full writeup covers the reported attack chain, backend control failures, and AI-specific blast radius.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.