Security teams should validate AI systems the way attackers use them, not just with static scans. That means testing prompt injection, tool execution chains, authorization boundaries, and data exposure paths in live workflows. The goal is to find where the model behaves as intended but the surrounding application or permission model still allows unsafe outcomes.
Why This Matters for Security Teams
AI copilots and AI-generated applications change the test target: the model may answer correctly while the surrounding workflow still leaks data, executes unsafe actions, or grants more access than intended. Security teams need exploitability testing that reflects real production paths, not only static analysis or benign prompts. That means testing tool use, identity boundaries, retrieval sources, and downstream business logic under adversarial conditions.
The risk is not theoretical. The NIST Cybersecurity Framework 2.0 emphasizes outcome-driven risk management, which fits AI systems that can act, call tools, and move data across trust boundaries. NHIMG research on the state of non-human identity security shows how often visibility and confidence lag behind real-world exposure, especially when machine identities and delegated access are involved. In practice, many security teams discover exploit chains only after a copilot has already been used to reach data or actions that were never meant to be reachable.
How It Works in Practice
Real-world exploitability testing should mirror the attacker path through the full stack: prompt surface, orchestration layer, tool permissions, data connectors, and auditability. The goal is to answer a simple question: what can the system actually do when an attacker shapes the conversation, context, or inputs?
Start with controlled adversarial exercises against the copilot or generated app in a production-like environment. Test prompt injection that tries to override system instructions, cause retrieval of sensitive documents, or coerce tool execution. Then verify whether the application enforces authorization at the point of action, not just at the point of user login. A model that cannot be “hacked” in isolation may still trigger a dangerous API call if the surrounding workflow trusts its output.
- Probe whether tool calls are constrained by least privilege and tenant-scoped access.
- Confirm whether secrets, tokens, and API keys are exposed to prompts, logs, or retrieved context.
- Test whether retrieved data can be exfiltrated through summaries, citations, or generated code.
- Validate whether approval steps exist for high-risk actions such as sending mail, creating tickets, or changing records.
Use production telemetry and policy enforcement to observe the same paths attackers would use. NIST CSF 2.0 is useful here because it frames testing around Identify, Protect, Detect, Respond, and Recover rather than around a single control family. For AI-specific abuse patterns, compare findings against DeepSeek breach analysis and the CoPhish OAuth Token Theft via Copilot Studio case to see how identity and delegation become the real exploit path.
These controls tend to break down when copilots are wired to broad enterprise connectors with inherited permissions, because the model can translate a harmless-looking prompt into an authorized high-impact action.
Common Variations and Edge Cases
Tighter testing often increases operational overhead, requiring organisations to balance faster AI delivery against slower but safer release gates. That tradeoff matters most when teams are generating applications with code assistants or deploying copilots that touch regulated or customer-facing data.
Current guidance suggests treating some AI-generated applications as continuously changing attack surfaces. If the code is produced or modified frequently, a one-time assessment is not enough; test the generated output, the surrounding runtime permissions, and the integration points after each meaningful change. Best practice is evolving, but there is no universal standard for this yet.
Edge cases also matter. A copilot that only drafts text may still be exploitable if it can read classified context and leak it into logs or exports. An AI-generated app may pass traditional SAST and DAST checks but still fail under conversational abuse, where the prompt becomes the payload and the tool chain becomes the exploit path. NHIMG’s State of Secrets in AppSec research is a reminder that secrets handling failures remain common, and those weaknesses become more dangerous when an AI system can surface or reuse them at scale. The practical lesson is to test for what the system can be made to do, not just what it was designed to do.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, CSA MAESTRO and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A2 | Prompt injection and tool abuse are core agentic exploitability tests. |
| CSA MAESTRO | AI-04 | MAESTRO covers agent control validation and runtime abuse paths. |
| NIST AI RMF | GOVERN | AI RMF governance supports risk-based testing and accountability. |
| NIST CSF 2.0 | PR.AC-4 | Least-privilege access controls are central to preventing unsafe tool actions. |
| OWASP Non-Human Identity Top 10 | NHI-03 | AI apps often fail through exposed secrets, tokens, and delegated credentials. |
Test agent workflows for prompt injection, tool chaining, and unsafe autonomous actions before release.
