TL;DR: Static scans and single-response tests miss AI failures that emerge at runtime, where prompts, memory, tools, and model interpretation can trigger unauthorized actions or data retrieval, according to OXSecurity. The real security test is continuous behavioural validation tied to execution paths, not output-only checks.
At a glance
What this is: OXSecurity argues that AI security testing fails when teams judge only the model output, because the real risk appears in runtime behaviour across prompts, memory, tools, and action selection.
Why it matters: IAM, PAM, and AI security teams need to validate what an agent is allowed to do at execution time, not only what it says, because hidden tool calls and data retrieval can bypass surface checks.
Context
AI security testing has a governance gap when teams validate only text output and ignore the runtime path that produced it. In AI-native systems, prompts, memory, retrieval, and tools all shape behaviour, which means a safe-looking response can still conceal unsafe execution. That matters for IAM, PAM, and NHI governance because the control boundary moves from the model answer to the action it is allowed to trigger.
The article frames a real problem for security operations and AI governance teams: static scanning and single-response checks assume deterministic behaviour, while AI systems change with context, phrasing, and history. For practitioners, the important question is whether the system respects policy when it reaches for data, calls tools, or expands scope beyond the original request. That is a runtime assurance problem, not a pre-deployment box-ticking exercise.
Key questions
Q: What breaks when AI security testing focuses only on prompt responses in isolation?
A: Testing only isolated prompts misses the ways adversaries actually manipulate AI systems in production. It can overlook indirect attacks through documents or retrieved content, and it may falsely suggest that a model is safe because it survives a narrow prompt set. The result is weak assurance, poor comparability, and an incomplete view of mission-specific risk.
Q: Why do AI agents increase IAM and PAM risk?
A: AI agents increase IAM and PAM risk because they can execute actions quickly once privilege is available, which shortens the time available to detect misuse. If access is always on, the attack surface is always on too. That is why task-scoped privilege and ownership controls matter.
Q: What are the signs that AI agent testing is too shallow?
A: If evaluation only checks refusal behaviour, overall benchmark scores, or end-to-end task success, it is probably missing the real attack surface. Shallow testing ignores whether the model can be pushed into tool misuse, data leakage, or indirect prompt injection at the exact decision point where failures occur.
Q: How should security teams govern AI agents that can access enterprise systems?
A: Security teams should govern AI agents as non-human identities with explicit ownership, scoped privileges, and continuous monitoring. The control set should include inventory, task-bound credentials, audit trails, and revocation paths. If an agent can call tools or touch production systems, it belongs in the same governance model as service accounts and other machine identities.
Technical breakdown
Why output-only testing misses AI runtime abuse
Output-only testing assumes the response is the security boundary. In AI systems, that assumption breaks because the model can appear compliant while still attempting unsafe tool calls, retrieving restricted data, or carrying hidden context into later steps. The real control surface is the execution chain that links prompt interpretation to tool selection and parameter construction. A model may refuse on the surface but still expose dangerous behaviour through retrieval or multi-turn steering. Security testing therefore has to inspect actions, not just strings, and compare those actions to the intended policy boundary.
Practical implication: Test the full execution path, including tool calls and retrieved data, not only the final answer.
How prompt, memory, and tool layers create hidden attack paths
AI applications distribute logic across several runtime layers. Prompts define instruction hierarchy, memory preserves prior context, and tools turn model decisions into external actions. Each layer can be influenced independently, which is why a single interaction can poison later behaviour or cause broader-than-intended access. If untrusted content enters shared memory or a prompt chain, subsequent steps may treat it as trusted. If tool parameters are inferred from adversarial phrasing, the model can expand scope without any code change. Traditional AppSec rarely models that layered behaviour, but AI testing must.
Practical implication: Review how instructions, memory, and tools interact, and isolate untrusted input before it reaches shared runtime context.
Why behavioural validation must map to access and data exposure
A behavioural failure only becomes a security incident when it can reach real assets. That means the interesting question is not whether the model made a questionable suggestion, but whether it could trigger a privileged action, query a sensitive database, or influence an internal workflow. Mapping test results to the access graph separates noise from exploitable risk. This is where AI security intersects with IAM and NHI governance: the permissions behind the agent matter as much as the model output. Without scope control at the tool layer, behavioural drift turns into access drift.
Practical implication: Tie AI test results to the underlying permission graph so unsafe behaviour is ranked by actual exposure.
Threat narrative
Attacker objective: The attacker aims to make the AI system perform an unauthorized action or retrieve sensitive data while the visible response still appears acceptable.
- Entry occurs through crafted prompts, multi-turn context, or injected instructions that influence how the model interprets the request.
- Credential or privilege abuse follows when the model selects tools or parameters that exceed the intended scope of access.
- Impact occurs when the resulting execution reaches sensitive data, internal systems, or privileged workflows that should not have been exposed.
Breaches seen in the wild
- DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.
- 12,000 secrets in LLM training data: Truffle Security found 11,908 live API keys and passwords hard-coded in web pages captured by Common Crawl, a dataset used to train LLMs.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Runtime abuse is the real AI security boundary: output checks are a weak proxy for trust when the system’s actual risk lives in tool invocation, retrieval, and memory state. The model can produce a benign answer and still cross an authorization line behind the scenes. For AI governance, the control question is whether the runtime path stayed inside the approved action set. Practitioners should treat execution as the security object, not the sentence returned to the user.
Decision integrity is the named control gap here: AI testing fails when teams validate content but not the path from intent to action. That gap is especially important where an AI system acts as a non-human identity through tools and APIs, because privilege is exercised through machine execution rather than human approval. This is where NHI governance and AI security converge. The practical conclusion is that authorization has to be enforced at the point of action, not inferred from output quality.
Hidden runtime abuse creates policy drift faster than static controls can see it: prompts, memory, and model updates can change behaviour without any code change. That means assurance evidence must be continuous, not a one-time pre-release gate. Security teams should expect behavioural variance and build controls that measure it directly. The field should stop treating AI safety as a model property and start treating it as an access and execution property.
AI security testing needs an access graph, not a test report: findings only matter when they are tied to reachable systems and sensitive data. That shifts prioritisation from generic prompt failures to the execution paths that can actually move information or trigger privileged state changes. The most mature programmes will combine behavioural testing with identity-aware authorization and runtime observability. That is the point where AI security becomes governable rather than merely observable.
Continuous behavioural validation is now a governance requirement: systems that learn, retrieve, and act cannot be judged by static snapshots of their output. Security leaders should assume that runtime behaviour will drift and that hidden abuse will surface first at the action layer. That means AI assurance, PAM discipline, and NHI controls have to be designed together. Practitioners should align their control model to execution, not just evaluation.
From our research library:
- 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
- Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
- Read next: AI Agent Observability, Audit and Incident Response Guide
What this signals
Decision integrity is the control gap AI security teams are now chasing: if a model can appear safe while taking unsafe actions behind the scenes, the old habit of judging only the output is no longer defensible. Continuous validation needs to follow the execution path, because that is where the real trust decision is made. The practical move is to instrument tool calls and retrieval so policy enforcement can be checked at the point of action.
AI security and NHI governance are converging at the runtime layer: when an assistant or agent can trigger APIs, it behaves like a machine identity with delegated authority. That does not make every model an NHI, but it does mean privilege management has to extend into the action layer. Teams that already run strong access controls should map them to AI execution paths before the next model update changes behaviour.
For practitioners
- Validate tool calls, not just model output Instrument AI systems so every prompt can be traced to the tools invoked, parameters passed, and data returned. Treat an unsafe call attempt as a control failure even if the final response is filtered.
- Enforce least privilege at the action layer Bind each agent or assistant to a narrow permission set for API access, retrieval, and write operations. The model should never be able to widen its own scope when it interprets a broader request.
- Test multi-turn and memory poisoning paths Run adversarial cases that seed misleading context, then check whether stored memory or shared prompt state carries that influence into later decisions. Multi-step abuse is where many runtime failures hide.
Key takeaways
- AI security testing fails when teams trust the final answer more than the hidden action path that produced it.
- Runtime behaviour, not static code, determines whether prompts, memory, and tools can reach sensitive systems.
- The strongest control is continuous behavioural validation tied to scoped permissions and observable execution traces.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | The article focuses on agents expanding or misusing delegated runtime authority. |
| ASI02 — Tool Misuse | The core failure is unsafe tool invocation and parameter handling during execution. | |
| Recommendation — Constrain agent privileges and verify every tool call against the intended authorization scope. Test agent tool paths for unsafe calls, broad parameters, and unintended side effects. | ||
| NIST AI RMF | MANAGE — AI Risk and Impact Management | Continuous behavioural validation and runtime oversight map directly to AI risk management. |
| Recommendation — Build runtime monitoring and red-team validation into the AI risk management cycle. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The article’s main control concern is whether AI systems stay within scoped access boundaries. |
| Recommendation — Apply authorization controls to AI actions so models cannot exceed granted permissions at runtime. | ||
| MITRE ATT&CK | TA0006;TA0010 — Credential Access; Exfiltration | The article describes AI paths that can expose data or sensitive material through runtime abuse. |
| Recommendation — Map runtime abuse scenarios to credential access and exfiltration tactics in detection engineering. | ||
Key terms
- Runtime abuse detection: Runtime abuse detection is the continuous evaluation of behaviour while a system is live, not after an incident review. For enterprise AI apps, it catches bot activity, credential abuse and suspicious access patterns before they become customer-visible failures.
- Tool Invocation: Tool invocation is an action where an AI agent calls an external system such as a database, API, or file service. Each invocation should be treated as an auditable identity action because it is the point where the agent can move data, trigger changes, or widen its reach across the environment.
- Decision Integrity: Decision integrity is the degree to which an organisation’s outputs remain consistent, explainable, and aligned to approved meaning. In data and AI programmes, it depends on controlled definitions, trusted sources, and the ability to prevent different tools from inventing conflicting interpretations.
- Session Trace: A complete record of one agent run from the initial prompt to the final output. It usually contains nested spans for model calls, tool invocations, and handoffs, giving teams a timeline that shows sequence, duration, and context across the full task.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and machine identity security. It helps security leaders connect delegated access, runtime control, and auditability across identity programmes.
Published by the NHIMG editorial team on August 2, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org