Notifications
Clear all
Topic starter
02/08/2026 11:32 am
TL;DR: Static scans and single-response tests miss AI failures that emerge at runtime, where prompts, memory, tools, and model interpretation can trigger unauthorized actions or data retrieval, according to OXSecurity. The real security test is continuous behavioural validation tied to execution paths, not output-only checks.
NHIMG editorial — based on content published by OXSecurity: AI security testing and runtime abuse in AI-native systems
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
Questions worth separating out
Q: How should security teams test AI systems beyond output filtering?
A: Security teams should test the full execution path, including prompt handling, retrieval, memory, tool selection, parameter construction, and downstream API responses.
Q: Why do AI agents complicate access governance more than ordinary automation?
A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context.
Q: What breaks when organisations rely on static scans for AI security?
A: Static scans miss runtime behaviour.
Practitioner guidance
- Validate tool execution paths, not just model responses Instrument the agent runtime so every tool call, parameter set, and downstream response is captured and reviewed against the intended task scope.
- Bind each AI tool to a fixed permission scope Treat model-to-tool integration like a privileged delegation boundary.
- Separate trusted instructions from untrusted input Keep system prompts, retrieval content, and user text clearly segmented so injected instructions cannot inherit trust.
What's in the full article
OXSecurity's full article covers the operational detail this post intentionally leaves for the source:
- Execution-trace examples showing how unsafe tool calls emerge from prompt variation and context poisoning
- Code snippets and workflow patterns for separating system prompts from untrusted user input
- Operational guidance for tracing model actions back to the actual access graph and sensitive data paths
- Testing scenarios that distinguish harmless output issues from exploitable runtime abuse
👉 Read OXSecurity's analysis of AI security testing and runtime abuse →
AI security testing and runtime abuse: are your controls keeping up?
Explore further
02/08/2026 12:46 pm
AI security testing is really identity-and-authorisation testing for the model runtime. The article makes clear that the highest-risk event is not a wrong answer, but a wrong action. Once a model can call tools, fetch data, or trigger workflows, the security question shifts to who or what is authorised to act, under what scope, and with what evidence. That is a governance problem as much as a testing problem, and it should be treated as such.
A question worth separating out:
Q: What should teams do when an AI system can call privileged tools?
A: Teams should apply least privilege at the tool layer, require strong parameter validation, and log every action to an auditable identity trail. The model should never be able to widen its own scope based on phrasing or conversation context. If a tool can export data, delete records, or modify accounts, its permissions need explicit governance.
👉 Read our full editorial: AI security testing fails when output checks miss hidden runtime abuse
