Notifications
Clear all
Topic starter
30/07/2026 10:47 pm
TL;DR: Modern LLMs learn role separation, instruction hierarchy, and prompt templating through control tokens and system prompts, but those same mechanisms can be abused through control token injection, fake context resets, and XML prompt spoofing, according to HiddenLayer. The security boundary is probabilistic, not absolute, so AI teams need layered runtime controls around the model, not prompt engineering alone.
NHIMG editorial — based on content published by HiddenLayer: Inside the Prompt: How LLMs Learn Roles, Follow Instructions, and Get Exploited
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
- 96% of technology professionals identify AI agents as a growing security threat, and 66% believe this risk is immediate.
Questions worth separating out
Q: How should security teams reduce prompt injection risk in agentic AI systems?
A: Security teams should treat all external text as potentially malicious instruction content and place controls outside the model.
Q: Why do prompt templates create security risk in LLM deployments?
A: Prompt templates create risk because they establish predictable structure that attackers can imitate or corrupt.
Q: What breaks when a model can be persuaded to treat untrusted text as system-level instruction?
A: What breaks is the assumed hierarchy between trusted policy and untrusted input.
Practitioner guidance
- Treat prompt inputs as untrusted data Inspect system prompts, retrieved documents, emails, and repository files as potential instruction carriers.
- Enforce runtime policy outside the model Use permission boundaries, tool allowlists, and approval gates for actions that touch sensitive systems.
- Log model context and tool decisions Record which inputs entered the context window, which role markers were present, and which tools the agent selected.
What's in the full article
HiddenLayer's full research covers the operational detail this post intentionally leaves for the source:
- Token-by-token examples of how ChatML-style control tokens are encoded and abused in practice.
- Concrete prompt spoofing patterns, including fake context resets and control-token injection examples.
- The article's examples of XML-style tag manipulation and how model behaviour changes when trusted sections are closed or replaced.
- Additional analysis of how indirect prompt injection travels through documents, emails, and repository files.
👉 Read HiddenLayer's analysis of LLM roles, instruction hierarchy, and prompt injection →
Prompt injection and role control: are your AI controls keeping up?
Explore further
30/07/2026 10:53 pm
Prompt injection is a privilege problem, not a language problem. The article shows that attackers do not need to break the model to influence it; they only need to persuade it that untrusted text belongs in a higher instruction tier. That matters because role-aware LLMs turn natural language into a de facto policy surface. Practitioners should treat prompt integrity as an access-control issue, not a content-moderation issue.
A few things that frame the scale:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%), according to AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to the same SailPoint research.
A question worth separating out:
Q: Who is accountable when prompt injection leads an AI agent to misuse tools or data?
A: Accountability sits with the organisation that granted the model access and failed to constrain its runtime behaviour. Security, IAM, application, and AI owners all share responsibility for the control plane around the model. If tool use is possible, governance must cover authorisation, monitoring, and incident reconstruction before deployment.
👉 Read our full editorial: LLM role control and prompt injection reshape agentic AI risk
