McKinsey Lilli shows how appsec flaws reshape AI assistant behavior

At a glance

What this is: This is an analysis of how appsec weaknesses around API exposure, SQL injection, and broken object-level authorization can change an AI assistant's behaviour without directly compromising the model itself.

Why it matters: It matters because identity, access, and governance teams increasingly have to secure the data paths and authorization layers behind AI assistants, not just the model layer users see.

By the numbers:

• 72 percent of the firm was active on the platform.

Context

McKinsey's Lilli incident is best understood as an application-security failure that reached an AI assistant, not as a model jailbreak. The public record points to exposed API documentation, unauthenticated endpoints, SQL injection, and cross-user access, which means the core failure sits in the software and access layers around the assistant.

For identity and access practitioners, the important lesson is that AI assistants often inherit the governance weaknesses of the systems underneath them. If prompts, routing rules, retrieval metadata, and user history are stored as mutable backend data, then ordinary application access can alter assistant behaviour without changing the model code.

The article also shows why AI assistants must be treated as governed application surfaces, not as isolated chat experiences. Once backend writes can influence what the assistant retrieves or says, access control, object-level authorization, and configuration governance become part of AI security whether teams label them that way or not.

Key questions

Q: What breaks when an AI assistant's backend data is writable by ordinary application access?

A: The assistant stops being a controlled interface and starts reflecting whoever can alter its prompts, routing rules, retrieval metadata, or user context. In that situation, database access can change output behaviour without changing the model itself, so security teams have to govern the state that feeds the assistant, not just the chat layer.

Q: Why do exposed API routes and unauthenticated endpoints matter so much for AI assistants?

A: Because they often sit in front of the real control plane for assistant state. If a backend route can read or write prompts, retrieval settings, or object records without proper authentication, the model inherits that exposure indirectly. The risk is not only data theft, but also unauthorized changes to what the assistant can say or see.

Q: How should teams test broken object-level authorization in AI assistant workflows?

A: They should test every object the assistant can access, not just the login path. That means checking conversation history, linked documents, user-specific records, and retrieval objects to confirm the caller is authorized for each one. If object checks are inconsistent, the assistant can blend another user's context into an apparently normal response.

Q: When does SQL injection become an AI governance problem instead of just an AppSec issue?

A: It becomes an AI governance problem when the database stores instructions, routing logic, retrieval metadata, or context that determines assistant behaviour. At that point, a successful write can alter what the model retrieves or outputs. Security teams should treat those database paths as behaviour-control surfaces, not as ordinary data stores.

Technical breakdown

Exposed API surfaces create the first foothold

Public API documentation and unauthenticated routes remove the first layer of control around an assistant. Once an endpoint is reachable without the intended authentication middleware, an attacker or researcher can reach backend functions that were supposed to be internal. In this case, the reported exposure was not the model itself but the application surface that fed it. That distinction matters because AI assistants are increasingly composed of ordinary web services, and those services determine what the assistant can see, write, or retrieve. If the API layer is weak, the model inherits that weakness indirectly.

Practical implication: inventory assistant-facing routes and enforce authentication on every backend function that can read or change assistant state.

SQL injection turns data writes into behaviour changes

SQL injection is a data-layer flaw, but in assistant architectures it can become a behaviour-control issue. If prompts, routing rules, retrieval settings, or user context are stored in database rows, a successful write path can alter what the assistant later uses at inference time. The article's key insight is that the assistant need not be directly manipulated in prompt space if the database stores the instructions the model will consume. That makes SQL construction around dynamic identifiers, JSON keys, and sort fields especially dangerous, because those elements are often concatenated rather than safely bound.

Practical implication: review every database path that writes assistant instructions, metadata, or retrieval policy as a potential behaviour-changing control surface.

Broken object-level authorization expands the blast radius

Broken object-level authorization, often shortened to BOLA, occurs when a system accepts an object identifier but fails to verify that the caller is allowed to access that record. In assistant systems, that can expose another user's history, knowledge objects, or linked client data even when the model itself is behaving normally. The problem is not just data theft. Once the assistant can pull in another person's context, its outputs can incorporate information that the caller should never have seen. That turns authorization failure into a response-integrity issue as well as a privacy issue.

Practical implication: test assistant workflows for object-level access control across every record the model can read or reference.

Threat narrative

Attacker objective: The objective is to reach assistant-controlled data and metadata so the system can be made to reveal or use information across user boundaries without a direct model exploit.

1. Entry appears to have started with public API documentation and unauthenticated endpoints that exposed backend functions to inspection and use.
2. Credentialless access and injection opportunity then allowed unsafe SQL construction to touch database paths that stored assistant-related data and metadata.
3. Cross-user access followed when object-level authorization failed, expanding the reachable data and the assistant's potential response scope.
4. The impact was not model compromise but behaviour manipulation through backend state that could change what the assistant retrieved or returned.

Breaches seen in the wild

• Salesloft OAuth token breach: hackers stole OAuth tokens to access Salesforce data via Salesloft.
• Canvas Instructure Data Breach: ShinyHunters exploits Canvas LMS platform to expose millions of student records via third-party NHI credential abuse.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Most enterprise AI incidents will still be software incidents first: The public record on Lilli shows that exposed endpoints, SQL injection, and broken object-level authorization can shape assistant behaviour without touching the model weights. That means the governing problem is often the application and identity layer surrounding the model, not the model itself. Teams that separate AI governance from application security are missing the real attack surface.

Prompt, routing, and retrieval data are identity assets when they are mutable: If backend rows determine what an assistant says or sees, then those rows are effectively privileged control data. Ordinary database access becomes a trust boundary crossing because it can change instructions, context, and permission scope at runtime. Practitioner implication: treat assistant state as governed security data, not as harmless configuration.

Broken object-level authorization is the hidden multiplier in AI assistant design: BOLA does more than leak a record. In assistant workflows it can blend another user's context into a seemingly legitimate response, which makes the governance failure visible only after the output is consumed. The implication is that access control for assistants must be checked at the object and workflow level, not only at login.

Behavioural blast radius: When prompt data, retrieval metadata, and routing logic live in ordinary application storage, a single database write can alter many downstream responses. That is not model compromise in the classic sense. It is a governance collapse where the assistant's output becomes a reflection of whoever can change its backend state, so practitioners must secure state transitions, not just prompts.

AI security programmes need a stronger join between IAM and AppSec: The Lilli pattern shows that assistant security is now a control stack problem. IAM governs who can reach the backend, AppSec governs whether the backend can be manipulated, and data governance governs what the assistant can disclose. Practitioner implication: align those disciplines around the assistant's real trust boundaries, not around the chat interface.

From our research library:

• 92% of organisations expose NHIs to third parties, raising concerns about supply chain security, according to the Ultimate Guide to NHIs.
• Secrets management is a top five cybersecurity priority for only 33% of organisations, behind cloud security (45%), API security (42%), and endpoint security (36%), according to the 2024 State of Secrets Management Survey.
• Read next: IAM and IGA Basics

What this signals

Behavioural blast radius: When prompt data, retrieval metadata, and routing logic are stored as mutable application data, one write can reshape many assistant responses. That shifts the control problem from model tuning to governed state change, which is why AppSec and IAM must be designed together for AI systems.

AI assistant governance is now an access-control problem as much as a model-risk problem. Teams should expect more incidents where the visible failure is in the assistant, but the root cause sits in ordinary software paths that were never hardened for privileged, identity-linked data.

The practical boundary is simple: if a backend object can change what an assistant sees or says, it needs the same scrutiny as any other privileged system state. That includes change control, object-level authorization, and traceability for who altered the data and why.

For practitioners

• Audit assistant-facing endpoints Map every route that can read or write assistant state, including undocumented and internal-only paths, and require authentication on all of them.
• Treat prompt and retrieval state as privileged data Classify prompts, routing rules, retrieval metadata, and user-history tables as security-sensitive objects with change control and access review.
• Test for BOLA across assistant workflows Verify that every object the assistant can return is checked against the caller's access rights, including conversation history and linked knowledge records.
• Review SQL paths that use dynamic identifiers Inspect queries that concatenate JSON keys, field names, table names, or sort parameters, and remove any path that can be influenced by user input.
• Separate assistant behavior control from model control Define who can change backend data that affects assistant responses, then align that authority with the same governance used for other privileged application settings.

Key takeaways

• The Lilli incident shows how exposed API surfaces, unsafe SQL paths, and broken object-level authorization can alter AI assistant behaviour without compromising the model directly.
• The public record supports a familiar application-security chain, not a model jailbreak, which is the more important lesson for identity and governance teams.
• Assistant state, including prompts, routing rules, and retrieval metadata, should be treated as privileged application data because backend writes can become output changes.

Key terms

• Broken Object-Level Authorization: A failure to check whether an authenticated identity may access a specific object, record, or device. The request succeeds because the credential is valid, but the application does not enforce per-object entitlement. In NHI environments, this turns a legitimate token into cross-resource exposure.
• Assistant State: Assistant state is the backend data that influences an AI assistant's behaviour, including prompts, routing rules, retrieval metadata, and user context. When that state is stored in ordinary application tables, it becomes a privileged control surface that must be governed like any other security-sensitive configuration.
• Behavioral Blast Radius: Behavioral blast radius is the amount of damage a risky action can cause once it intersects with access, privilege, or sensitive data. The concept helps teams separate everyday user mistakes from incidents that can materially affect systems, data, or operations.
• Application-Security Chain: An application-security chain is a sequence of ordinary software flaws, such as exposed endpoints, injection, and authorization gaps, that combine into a larger compromise. For AI systems, the chain often matters more than the model itself because the model inherits the security posture of the surrounding application.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.