Adversarial prompt engineering exposes the AI security gap in LLMs

At a glance

What this is: Adversarial prompt engineering is a class of LLM attack that uses crafted inputs to bypass safeguards, exfiltrate data, and force unintended behaviour.

Why it matters: It matters because IAM, PAM, and AI governance teams need to treat LLM interactions as controlled access paths, not just text prompts, when protecting data and decision workflows.

By the numbers:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.
• 17 minutes

👉 Read Obsidian Security's analysis of adversarial prompt engineering and LLM attack paths

Context

Adversarial prompt engineering targets the control layer that sits between users, applications, and large language models. In practice, it exploits the fact that many organisations still treat AI output as a content problem rather than an access problem, even when prompts can trigger data exposure, policy bypass, or unsafe actions. That is why prompt injection belongs in the same governance conversation as secrets, privilege, and auditability.

For security and identity teams, the issue is not simply whether an LLM can be tricked. The real question is whether the model has visibility, authorisation boundaries, and logging strong enough to prevent the prompt from becoming a covert access path into systems or data. That intersection with IAM and NHI governance is what makes this topic operationally important, not merely theoretical.

Key questions

Q: How should security teams stop prompt injection from turning into tool misuse?

A: They should enforce authorization at the tool or resource boundary, not inside the model. Scope each tool to the minimum necessary permission, then deny any call that falls outside that scope. Prompt filtering still has value, but it cannot be the last line of defence because the model is exactly what the attacker is trying to influence.

Q: Why do LLMs create risk in identity and access management?

A: LLMs create risk when teams confuse fluent answers with verified security evidence. A model can summarize access patterns quickly, but it can also hide missing context, stale data, or inaccurate scope. In IAM, that means every output needs traceability back to the underlying identities, entitlements, and events.

Q: What do organisations get wrong about protecting AI assistants from malicious prompts?

A: They often focus on filtering bad language while ignoring the authorisation layer around the model. That misses the real failure mode, which is a prompt causing the system to retrieve data, call a tool, or act with privileges that should have been separately constrained. Good defence starts with access design, not just content moderation.

Q: Who is accountable when an AI model exposes data after a prompt attack?

A: Accountability usually sits with the team that approved the model's access model, the data owners who exposed the content, and the security function that failed to monitor the workflow. Frameworks such as NIST AI RMF and identity governance practices help define ownership, but the organisation must make tool access, logging, and review responsibilities explicit.

Technical breakdown

How prompt injection manipulates model instruction hierarchy

Prompt injection works because many LLM applications blend user input, system instructions, and retrieved context into a single conversational stream. A malicious prompt can override or confuse those layers, especially when the application fails to separate trusted instructions from untrusted content. The model is not being 'hacked' in the classic sense. Instead, it is persuaded to prioritise attacker-supplied text over the developer's intended policy. In enterprise deployments, the risk increases when retrieval-augmented generation, plugins, or tool use are added without strict instruction boundaries.

Practical implication: isolate system prompts, retrieved content, and user input, then enforce allowlists on what the model can see and do.

Why jailbreaks and multi-turn attacks erode guardrails

Jailbreaks are structured attempts to induce the model to ignore safety constraints, often through role-play, reframing, or repeated nudging across several turns. Multi-turn attacks matter because each exchange can weaken guardrails incrementally, especially in systems that preserve conversational state. The danger is compounded when the model has access to enterprise data or tools, because a successful jailbreak is not just a policy failure. It can become an authorisation failure if the model can act on behalf of a user or service without separate verification.

Practical implication: treat multi-turn conversations as an attack surface and re-authorise sensitive actions before each tool call or data retrieval.

How identity controls determine whether an LLM becomes an access path

LLM applications often inherit the identity of the surrounding application, but the model itself may still be able to request tools, query data, or emit instructions that trigger downstream actions. This creates a governance gap when access is granted broadly to a service account or API key that the model can indirectly abuse. In NHI terms, the model is not the identity, but it can become an identity amplifier if its access is not constrained by workload identity, scoped tokens, and audit trails. That is where AI security and NHI governance converge.

Practical implication: bind LLM tool access to scoped workload identities and review every privileged integration as if it were an NHI.

Threat narrative

Attacker objective: The attacker wants to turn the model's trust in language into unauthorised access to data, decisions, or connected systems.

1. Entry occurs when an attacker submits a crafted prompt, hidden instruction, or poisoned conversational context to an exposed LLM workflow.
2. Escalation occurs when the model follows attacker instructions, reveals protected data, or invokes connected tools with the application's own privileges.
3. Impact occurs when the attacker extracts sensitive information, violates policy, or uses the compromised AI workflow as a stepping stone into downstream systems.

NHI Mgmt Group analysis

Prompt injection is now an authorisation problem, not just a content moderation problem. Once a model can access tools, records, or workflows, the question is no longer whether it generates unsafe text. The real issue is whether untrusted language can trigger privileged behaviour. That shifts the governance burden toward IAM, logging, and tool-scoped controls, because the model's output may become an indirect control signal for downstream systems.

AI applications expose a new form of NHI sprawl: model-adjacent identities. Service accounts, API keys, and orchestration tokens often sit around the model with broader access than the business case requires. This creates a named failure mode we can call model-to-tool privilege spillover, where an LLM inherits more authority than its task needs. Practitioners should treat every model connector as a governed identity boundary, not a convenience integration.

Guardrails fail when teams assume the model itself is the control plane. LLMs do not reliably distinguish malicious from legitimate instructions unless the surrounding architecture enforces that separation. That means policy cannot live only in prompts, content filters, or fine-tuning. Security teams need layered controls that include tool authorisation, audit logging, and retrieval scope limits, otherwise prompt engineering becomes a bypass technique for the whole AI stack.

AI governance is converging with identity governance faster than many programmes have planned for. The article shows why enterprise AI security cannot be separated from workforce IAM, NHI management, and PAM discipline. Once models can retrieve data or call functions, the same questions recur: who approved the access, what can it reach, and how is misuse detected. The practical conclusion is that AI security review must sit inside identity governance, not beside it.

What this signals

Model-to-tool privilege spillover will become a recurring failure mode as enterprises connect LLMs to data, ticketing, code, and workflow systems. The security question is no longer whether the model can be manipulated, but whether the surrounding identity architecture prevents manipulation from turning into action. Practitioners should align AI controls with NIST AI Risk Management Framework principles and tool-scoped authorisation.

AI security programmes will need to distinguish between content safety, access safety, and execution safety. Those are different control problems, and they need different telemetry. A model that refuses harmful text can still leak information or trigger a privileged tool call if the workflow is poorly designed, so security teams should measure whether each AI action is separately authorised and logged.

The more agents and assistants enterprises deploy, the more identity governance will resemble workload governance. That makes NHI discipline relevant well beyond traditional service accounts. Teams that already manage OWASP NHI Top 10 style risks will be better positioned to govern agentic systems as they move from isolated pilots into production.

For practitioners

• Define model tool boundaries as identity boundaries Inventory every LLM connector, plugin, and API integration, then assign a scoped workload identity to each one with the minimum permissions needed for the task. Review shared service accounts and overbroad tokens first, because those are the easiest paths for prompt-driven abuse.
• Separate untrusted input from trusted instructions Implement strict prompt segmentation so user text, retrieved content, and system policy are handled differently. Block the model from treating external content as instructions, and test for prompt injection in retrieval-augmented generation workflows, chat history, and uploaded documents.
• Log every sensitive model action Capture prompts, retrieved documents, tool calls, and resulting outputs in an audit trail that security and compliance teams can review. Focus on actions that touch customer data, internal policies, or privileged workflows, because those are the events most likely to matter in investigations.
• Red-team the AI workflow, not just the model Test how the application behaves when attackers use multi-turn persuasion, hidden instructions, and malicious retrieval content. Include downstream systems in the exercise so you can see whether a successful prompt manipulation would also trigger data access, record updates, or workflow execution.

Key takeaways

• Adversarial prompt engineering exploits the gap between what a model reads and what a system is authorised to do.
• The evidence points to a governance problem as much as a technical one, because model access often expands through weak identity boundaries and incomplete logging.
• Practitioners should treat AI tool access like NHI governance, with scoped identities, segmented inputs, and audit-ready action trails.

Key terms

• Prompt Injection (Agentic): An attack where malicious instructions are embedded in content that an AI agent reads — causing the agent to execute unintended actions using its own legitimate credentials. A primary vector for agent goal hijacking and identity abuse.
• Jailbreak: A prompt or input transformation that bypasses a language model’s safety restrictions and causes it to produce output it would normally refuse. In operational settings, a jailbreak matters because the model may be embedded in a workflow, making the bypass a pathway to broader misuse, not just a bad response.
• Model-To-Tool Privilege Spillover: Model-to-tool privilege spillover occurs when an LLM can influence downstream systems through credentials, plugins, or orchestration paths that are more powerful than the task requires. The model becomes a privilege amplifier because its conversational output can indirectly cause privileged action.
• Adversarial Red-Teaming: Adversarial red-teaming is the practice of actively trying to make a security model fail before it reaches production. The test uses crafted inputs, boundary probes, and mutation strategies to reveal weaknesses that ordinary accuracy testing will not show.

What's in the full article

Obsidian Security's full blog post covers the operational detail this post intentionally leaves for the source:

• Walkthroughs of specific prompt injection and jailbreak patterns that defenders can reproduce in testing
• Step-by-step mitigation methods for input validation, output filtering, and behavioural monitoring
• Implementation guidance for zero-trust controls around AI tool access and audit logging
• Operational examples showing how teams can integrate AI incident response into existing SOC workflows

👉 The full Obsidian Security post covers prompt attack patterns, mitigation controls, and implementation guidance in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It is designed for practitioners who need to connect identity controls to broader AI and security programmes.