Prompt injection is exposing the limits of legacy AI security

At a glance

What this is: This article argues that prompt injection is the AI-era analogue of SQL injection, but with a broader and harder-to-control attack surface because instructions and data are merged.

Why it matters: It matters because IAM, NHI, and AI governance teams now need runtime controls, tool governance, and visibility for AI systems that can act on instructions embedded in ordinary content.

👉 Read AppSOC's analysis of prompt injection and AI agent security

Context

Prompt injection is a governance problem as much as a technical one. It works because LLMs and AI agents do not reliably separate trusted instructions from untrusted content, so malicious text can alter behaviour inside workflows, tools, and data pipelines. For identity and access teams, that creates a new control challenge around who or what is allowed to influence agent decisions.

The article frames prompt injection as a repeat of the early SQL injection era, but with wider blast radius because AI systems ingest emails, PDFs, web pages, and tool outputs. In practice, that means the boundary between content and command is now a security perimeter, and it is often unprotected.

Key questions

Q: How should security teams govern AI agents that can change behaviour based on prompt context?

A: Treat the agent as a runtime identity, not a fixed script. The control goal is to limit how untrusted context can change tool use, output shape, and execution path. That means separating instructions from user data, constraining outputs into structured fields, and validating every tool boundary before the agent can act.

Q: Why is indirect prompt injection harder to defend than XSS?

A: XSS is usually defeated by deterministic controls such as sanitisation and output encoding. Indirect prompt injection is harder because the model interprets natural language non-deterministically and may turn untrusted text into different actions depending on context, so the security boundary must sit around the model, not inside it.

Q: What breaks when AI tools are exposed through loosely governed MCP servers?

A: Loose governance lets model-driven tools cross from context retrieval into state-changing actions without enough oversight. That can expose sensitive data, trigger unauthorized system changes, or widen lateral movement paths. The failure is a control boundary mismatch between what the AI can ask for and what it can safely do.

Q: How do teams know whether prompt injection controls are actually working?

A: Look for end-to-end visibility across prompts, retrieved content, memory, tool calls, and outputs, plus evidence that blocked actions stay blocked under realistic test cases. If the system can only be evaluated with static prompts, the controls are probably too narrow. Behaviour drift under multi-turn workflows is the signal to watch.

Technical breakdown

Why prompt injection breaks the instruction and data boundary

Prompt injection succeeds because the model treats all text as part of one conversational context. Unlike SQL engines, LLMs do not enforce a hard separation between code and data, so malicious instructions can be hidden inside ordinary content such as documents, web pages, or chat messages. In agentic systems, that context can flow into tool calls, memory, and downstream actions. The result is not just bad answers but compromised behaviour across connected systems.

Practical implication: teams need policy and runtime controls around where AI inputs come from and what contexts agents are allowed to trust.

Why indirect prompt injection is more dangerous than simple jailbreaks

Direct jailbreaks target the user prompt, but indirect prompt injection hides malicious instructions in third-party content the model later retrieves or reads. That makes the attack path harder to spot because the user never sees the payload and security teams may not know which source carried it. Once the model incorporates that hidden instruction, the compromise can influence summarisation, classification, tool use, or agent delegation. The operational issue is that trust now depends on content provenance as much as user identity.

Practical implication: organisations need source validation and provenance-aware controls for retrieved content, not just prompt filters.

Why AI agents and MCP connections expand the attack surface

When an LLM can invoke tools, the prompt becomes a control input to real systems. MCP connections, plugins, APIs, and agent workflows turn a text manipulation problem into an execution problem, because a successful injection can trigger actions outside the model itself. That shifts risk from conversational misuse to privilege misuse. In identity terms, the AI agent becomes a non-human identity whose authority must be bounded, audited, and revoked like any other machine actor.

Practical implication: apply least privilege, allowlisting, and continuous monitoring to every tool and MCP connection an agent can reach.

NHI Mgmt Group analysis

Prompt injection is becoming the defining governance failure of agentic AI. The issue is not simply that models can be tricked. It is that enterprises are granting AI systems decision influence before they have defined the trust boundaries those systems need. That makes prompt injection a control-plane problem for AI governance, not a content-filtering problem. Practitioners should treat it as a new class of runtime authorisation risk.

AI agents must be governed as non-human identities, not as passive software features. Once an agent can select tools, time actions, or chain steps through MCP and API calls, it behaves like a machine actor with delegated authority. That means the core questions become identity, privilege, and auditability. The absence of explicit lifecycle controls for agent permissions creates the same governance gap seen in unmanaged service accounts, only with faster and less predictable behaviour. Practitioners should align AI agent governance with NHI controls.

Prompt injection exposes the failure of static security assumptions in dynamic systems. Traditional controls assume that bad input can be detected, blocked, or sanitised before execution. The article shows why that model breaks when the system itself interprets language probabilistically and can act on it. That leaves organisations with a verification trust gap between what was intended and what the model did. Practitioners should move from input screening to continuous behavioural verification.

Shadow AI will make prompt injection a discovery problem before it becomes an enforcement problem. If organisations cannot inventory every model, agent, tool, and embedded AI feature, they cannot meaningfully secure them. This is the same visibility problem that has historically weakened NHI governance, only now the estate includes hidden assistants, unmanaged workflows, and third-party integrations. Practitioners should build AI usage discovery into their control baseline before expanding deployment.

Prompt injection will push the market toward AI-native security controls, but governance maturity will still decide outcomes. Tooling that tests, monitors, and blocks suspicious behaviour is necessary, but it cannot compensate for unclear ownership, weak approval models, or excessive agent privilege. The organisations that reduce exposure fastest will be the ones that connect AI security to IAM, PAM, and change governance. Practitioners should expect security architecture to converge around that operating model.

What this signals

Prompt injection will force security programmes to move from content filtering to behavioural control. The practical shift is from trying to spot bad text to controlling what models and agents are allowed to do after they read it. For identity teams, that means agent permissions, approval paths, and runtime telemetry become part of the security baseline rather than optional hardening.

Shadow AI discovery should be treated as a prerequisite control, not a reporting exercise. If you cannot see every model, assistant, and agentic workflow, you cannot protect them or assign accountability. That is especially true where AI systems inherit access to sensitive data or enterprise tooling.

AI agent governance will converge with NHI governance because both problems are about delegated machine authority. The same questions now apply across service accounts, bots, and AI agents: who owns it, what can it do, how is it audited, and when is access removed. The programmes that unify those controls will have the clearest path to measurable risk reduction.

For practitioners

• Define trust boundaries for every AI workflow Map which prompts, documents, web sources, and retrieved objects can influence each model or agent, then classify trusted and untrusted inputs separately. Use that mapping to decide where prompt injection controls and review steps must sit in the workflow.
• Restrict agent tool access to minimum necessary scope Treat every tool, API, and MCP connection as a privilege boundary. Allow only the actions an agent must perform, require explicit approval for sensitive operations, and remove broad default access from general-purpose AI assistants.
• Add runtime monitoring for suspicious AI behaviour Monitor actual model outputs, tool invocations, unexpected data access, and attempts to override system instructions. Use alerts for anomalous agent behaviour rather than relying only on pre-deployment testing.
• Inventory shadow AI across business units Discover embedded assistants, local copilots, and unsanctioned agent workflows across teams so you can apply governance before exposure grows. Without a complete inventory, prompt injection risk remains largely invisible.
• Test models against indirect injection paths Include PDFs, emails, scraped web pages, knowledge bases, and multimodal content in adversarial testing. The main failure mode is not only the user prompt, but the hidden instructions buried in upstream content.

Key takeaways

• Prompt injection is not just a model quality issue. It is a control failure that can turn ordinary content into unauthorised action.
• The scale of the problem is already visible in AI agent overreach, shadow AI, and the widening gap between deployment and governance.
• The most effective response is to govern AI agents as non-human identities with runtime monitoring, strict tool scoping, and lifecycle accountability.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Indirect Prompt Injection: Indirect prompt injection is an attack where malicious instructions are hidden inside content that an AI system reads later. The model may treat that content as context rather than as hostile input, which can influence tool use, data access, or workflow actions if controls are weak.
• Model Context Protocol: Model Context Protocol is an open protocol that lets AI agents connect to tools and data sources. It expands what an agent can reach, so governance has to cover not only the model and its prompts, but also every system that can receive or return agent-driven data.
• Shadow AI: AI agents, copilots, or connected tools operating without full visibility or governance from security teams. Shadow AI becomes an identity problem when those systems authenticate with unmanaged tokens, service accounts, or OAuth apps that can reach production resources.

What's in the full article

AppSOC's full blog post covers the operational detail this post intentionally leaves for the source:

• Adversarial testing patterns for direct, indirect, and multimodal prompt injection across model families
• Runtime guardrail examples for catching unsafe tool calls, data access, and system instruction overrides
• MCP and agent-tool governance details that show how to validate connections and reduce misuse risk
• AI usage discovery methods for identifying hidden models, assistants, and embedded workflows across the enterprise

👉 The full AppSOC article covers the SQL injection comparison, threat mechanics, and AI-native defence model in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners connect delegated access, lifecycle control, and auditability across human and machine identities.