How can security teams tell if an agent is being manipulated by hidden instructions?

Why This Matters for Security Teams

Hidden instructions are dangerous because they can turn an apparently routine agent into a covert data mover, policy bypasser, or privilege escalator. The risk is not limited to prompt text alone. It includes injected content in documents, web pages, tickets, or retrieved context that changes the agent’s behavior without altering its stated objective. Guidance from the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework both point to the same operational reality: teams need to assess agent behaviour, not just model output quality.

Security teams often get caught out because the agent still appears “successful” from a business perspective while quietly crossing trust boundaries. A summary agent may begin extracting more than requested, a support agent may follow embedded instructions from an untrusted page, or a workflow assistant may reframe a low-risk task into one that exposes secrets. The core problem is that hidden instructions rarely look obviously malicious in isolation. They depend on the agent’s willingness to treat retrieved content, user content, and system instructions as interchangeable inputs.

In practice, many security teams encounter this only after an agent has already disclosed data or acted outside scope, rather than through intentional policy testing.

How It Works in Practice

Detection starts with comparing the agent’s observed path to the approved task path. If an agent was asked to classify, summarise, or triage content, its tool use should remain tightly aligned to that purpose. Analysts should look for signs that the agent has accepted instructions from untrusted context, such as following embedded directives in a document, switching topics without a clear trigger, or escalating from read-only access to broader retrieval. This is consistent with the threat patterns described in the MITRE ATLAS adversarial AI threat matrix and the CSA MAESTRO agentic AI threat modeling framework.

• Inspect tool sequences for unusual jumps, such as retrieval followed by export, forwarding, or file creation.
• Check whether the agent cites or obeys instructions found inside retrieved content, especially when those instructions conflict with the task.
• Review context handling for repeated resets, re-prompts, or unexpected expansion of the working set.
• Validate whether the agent is asking for data it was not meant to access.
• Correlate suspicious runs with prompt logs, retrieval logs, and approval boundaries.

Good practice is to build tests that include benign-looking injected instructions inside sample pages, tickets, and attachments, then observe whether the agent resists them. Where an organisation uses agentic workflows for SOC enrichment, code review, or customer support, the monitor should also capture whether the agent is surfacing untrusted content as a control signal or as an action directive. Public reporting on real-world misuse, including the Anthropic report on AI-orchestrated cyber espionage, reinforces that operational misuse can emerge through ordinary task flows.

These controls tend to break down when agents are given broad tool access, weak provenance controls, and untrusted retrieval sources in the same execution path.

Common Variations and Edge Cases

Tighter detection often increases review overhead and false positives, requiring organisations to balance workflow speed against confidence in agent autonomy. Best practice is still evolving for agents that operate across multiple sessions or use long-lived memory, because the line between persistent context and covert instruction can be hard to prove from logs alone.

Some edge cases are especially difficult. A legitimate workflow may look suspicious if it intentionally moves from retrieval to summarisation to drafting to submission. Conversely, a manipulated agent may look normal if the hidden instruction is subtle, such as changing tone, enlarging scope, or prioritising disclosure only at the final step. The practical question is whether the behaviour is explainable from the declared task and the approved permissions model.

Where the environment includes external content ingestion, current guidance suggests treating untrusted text as data, not instruction, and isolating it from system-level control paths. That principle is central to the OWASP Top 10 for Agentic Applications 2026 and aligns with the governance approach in the NIST framework. Teams should also distinguish between agent failure and intended autonomy. A planning agent that explores options is not necessarily compromised; a planning agent that repeatedly seeks permission to ignore restrictions is.

In environments with high-volume retrieval, nested agents, or delegated tool chains, the distinction becomes harder because the instruction boundary is fragmented across services, making manipulation harder to spot and slower to contain.

Related resources from NHI Mgmt Group

• How can security teams tell whether AI agent access is drifting out of scope?
• How can security teams tell whether agent access is actually under control?
• How can security teams tell whether agent file access is drifting out of policy?
• How can security teams tell whether agent permissions are too broad?