They should assess what the system can see, retrieve, and act on, then test for prompt injection, data poisoning, and unsafe delegated actions. The key decision is to govern the AI system as a reachable asset with bounded authority, not as a passive model that only generates text.
Why AI Systems Should Be Tested Like Reachable Assets
When an AI system can retrieve internal content, call tools, or trigger downstream actions, it is no longer just a text generator. It has a real attack surface: prompts, memory, retrieval sources, tool permissions, and the policy layer that decides what it can do. Teams should scope the system as an operational asset with boundaries, dependencies, and failure modes, then test those boundaries directly.
That shift matters because the highest-risk failures are usually not model accuracy issues. They are authority and exposure issues: the system sees too much, can retrieve too much, or can act beyond what the business intended. A bounded-asset view forces teams to ask what data and actions are actually reachable, not what the model might produce in isolation.
For teams building this discipline, the NIST AI Risk Management Framework and the NIST AI 600-1 GenAI Profile are useful anchors because they emphasise governance, pre-deployment testing, and risk treatment for generative systems that influence real decisions.
What Teams Need to Test First
The first test is reachability. Identify what the system can see through prompts, retrieval, logs, memory, and connected knowledge sources. Then verify what it can retrieve on demand, including content that should remain hidden from ordinary users or unrelated workflows. If the AI can surface sensitive records, internal instructions, or privileged context, those exposures become part of the security boundary.
The second test is actionability. Map every tool, API, workflow, and delegated action the system can invoke, including write operations, approvals, ticket creation, file changes, and messages sent to other systems. The important question is not whether the model can describe a safe outcome. It is whether the runtime policy actually blocks unsafe or unintended execution paths.
The third test is trust in inputs. Prompt injection and data poisoning are not abstract model problems, they are control failures in the surrounding system. If the AI consumes untrusted content from documents, web pages, tickets, chat, or retrieval stores, that content can shape what the system says, what it retrieves next, and whether it attempts a tool call at the wrong time.
When the attack surface is built around retrieval and tools, the relevant testing discipline resembles the OWASP API Security Top 10 and the OWASP Web Security Testing Guide in spirit: enumerate trust boundaries, then probe for broken authorization, unsafe access paths, and input-driven abuse of downstream functions.
How to Govern Prompt Injection, Poisoning, and Delegated Actions
Prompt injection tests should focus on whether hostile instructions can override system intent, alter retrieval behaviour, or push the model into unsafe tool use. Teams should test both direct injection and indirect injection through retrieved content, because systems often trust indexed material more than they should. The objective is to prove that untrusted text cannot become hidden policy.
Data poisoning tests should examine whether corrupted or misleading content can persist in the knowledge base, retrieval index, fine-tuning set, or operational memory. The concern is not only bad answers. Poisoned content can steer the system toward wrong decisions, wrong escalation paths, or wrong actions while still appearing plausible to operators.
Unsafe delegated actions are the point where AI security becomes operational security. If the system can submit requests, update records, message users, or invoke higher-trust workflows, the policy should enforce least privilege, bounded scopes, and explicit approval for sensitive actions. This is especially important where a tool action has side effects that cannot be rolled back cleanly.
The strongest external references for this part of the problem are MITRE ATLAS adversarial AI threat matrix and the CSA MAESTRO agentic AI threat modeling framework, both of which help teams reason about prompt injection, tool misuse, and cascading agent behaviour.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and OWASP API Security Top 10 address the attack and risk surface, while NIST AI RMF and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Govern | AI systems with tools and retrieval need governance over reachable data and actions. |
| Recommendation — Define authority boundaries and test them before approving production use. | ||
| NIST SP 800-53 Rev 5 | AC-6 — Least Privilege | Delegated AI actions should be bounded by least privilege to limit unsafe execution. |
| Recommendation — Restrict tool and workflow permissions to the minimum required. | ||
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | The question centers on unsafe delegated actions by an AI system. |
| Recommendation — Test whether prompts can drive tools beyond intended policy. | ||
| MITRE ATLAS | Adversarial Machine Learning Threat Matrix | Prompt injection and data poisoning are core adversarial AI threat paths. |
| Recommendation — Map injection and poisoning cases to threat techniques during testing. | ||
| OWASP API Security Top 10 | API5 — Broken Function Level Authorization | AI-connected tools often fail when functions can be invoked beyond intended authority. |
| Recommendation — Verify every AI-invoked function is authorization-checked server side. | ||
Practitioner Guidance
What to prioritise: Start with the controls that change blast radius, not the controls that merely improve model quality. If a prompt can steer a write-capable workflow, that is a higher-priority problem than a harmless hallucination.
What to verify: Prove that the system cannot retrieve or expose material it should not see, and that every sensitive tool action is gated by explicit policy, not by prompt wording or user intent alone. If the policy cannot be demonstrated with test cases, it is not yet operationally trustworthy.
Decision rule: If an AI system can affect production data, operational workflows, or access decisions, treat it as a governed asset with bounded authority and test it like one. If it only drafts text with no connected actions, the testing emphasis should stay on content safety and information exposure.
Practitioner takeaway: The central mistake is to test the model as if it were isolated from its surrounding permissions. The real security question is whether the system can be manipulated into reaching, revealing, or doing more than its authority should allow.
Related resources from NHI Mgmt Group
- How should security teams limit the risk from AI agents that have access to production systems?
- How should security teams govern AI agents that can access enterprise systems?
- How should security teams govern machine identity credentials in agentic AI environments?
- How should security teams manage permissions for AI agents?