What do teams get wrong when they assess AI risk with model testing alone?

Why model testing is only one slice of AI risk

Model tests answer a narrow question: how the model behaves under prompts, stress, or adversarial input. They do not, by themselves, tell you what a system can reach, what it can change, or how far a bad output can travel once the model is embedded in a workflow, threat modelled as an agent, or connected to production data and actions. That is why model quality and system exposure are not the same control problem.

In practice, the gap appears when teams treat “the model passed tests” as evidence that the whole AI application is safe. A model can be robust to ordinary prompt attacks yet still sit inside a pipeline that lets it call tools, retrieve sensitive context, or trigger downstream actions. The exposure is in the surrounding system design, not only in the model weights or response text.

This is also where AI risk becomes broader than classic red-teaming. Model testing is useful for prompt sensitivity, jailbreak resistance, and instruction-following behaviour, but it does not prove safe delegation, bounded authority, or safe dependency handling. If the runtime can touch secrets, production paths, or privileged APIs, then the real question is whether the system constrains those paths under failure, not whether the model sounds well behaved in a lab.

What model-only testing misses in real deployments

Teams usually miss the parts of AI systems that sit between the prompt and the outcome. Those include tool permissions, retrieval scope, approval flows, session context, and the identity or credential used by the application component. When those pieces are not tested, prompt injection can become a path to unintended tool use, data access, or action execution even if the base model appears resilient.

That is why red teaming AI agents for identity abuse matters as a complement to model testing. It shifts attention from “can the model be persuaded?” to “what can the system do if persuasion succeeds?” The same logic applies to over-privileged agents, shared credentials, and runtime access to production systems. Those conditions create a blast radius that model-only evaluation will not reveal.

Teams also under-test the dependencies around the model. An AI system may call other services, consume retrieved content, or pass outputs into automation steps. If those dependencies are trusted by default, the model becomes a decision point inside a larger chain of trust. The result is that an apparently minor model failure can turn into data exposure, unauthorized actions, or corruption of downstream business logic.

How to assess AI risk as a system, not a demo

Effective assessment starts by mapping the full execution path: what the model can see, what it can call, what it can write, and what identity or secret lets it do those things. The unit of analysis should be the end-to-end workflow, including guardrails, tool brokers, approval gates, logging, and exception handling. If you do not test those boundaries, you are only evaluating one component of the risk surface.

For organisations building or buying agentic systems, agentic AI identity risk is a useful framing because it forces the right questions about authority, delegation, and accountability. Who can the agent act as? Which actions are reversible? Which secrets are reachable? Which production paths are gated by human review? Those are the questions that separate a safe pilot from a system that can fail in production.

Assessment should also distinguish between model confidence and operational safety. A model may produce accurate answers in testing while still being unsafe in context because the surrounding system lacks isolation, privilege boundaries, or robust input validation. The practical benchmark is not “did the model look good?” but “can a bad input or malicious prompt cause a material action, data leak, or unauthorized dependency call?”

Risk and Threat Considerations

Model-only testing creates a false sense of security because it measures language behaviour while leaving runtime authority untested. That is where prompt injection, tool abuse, and over-privileged access become exploitable, especially when the agent can reach secrets or production systems.

Failure mechanism: The model appears resilient in isolation, but the deployed workflow allows it to invoke tools, retrieve sensitive context, or execute actions with more privilege than intended, so an attacker only needs to influence the prompt path.

Impact: The result can be unauthorized data exposure, unsafe downstream actions, lateral movement through trusted integrations, or business process manipulation that testing never observed.

Practitioner Guidance

What to verify: Test the complete AI workflow, not just the model. Confirm which tools, secrets, data sources, and production paths are reachable from a successful prompt or retrieval chain, and verify the effective privilege of the runtime identity rather than the intended design.

Decision rule: If an AI component can take an action that would be unacceptable for a human operator with the same context, treat that as an access-control problem, not only a model-quality problem. If the action reaches production, prioritise blast-radius reduction before more prompt tuning.

What practitioners underestimate: Model testing often proves only that the system can be persuaded, not that it is bounded. The safest deployments are the ones where a compromised prompt still cannot reach sensitive credentials, privileged APIs, or irreversible actions.

Practitioner takeaway: AI risk assessment has to follow the authority chain, because the dangerous part of the system is usually the combination of model behaviour, runtime access, and downstream action, not the model output alone.

Related resources from NHI Mgmt Group

• What do teams get wrong when they rely on static analysis alone for AI model security?
• What do security teams get wrong when they assess AI pentesting tools by feature lists alone?
• What do teams get wrong about the EU Data Act when they assume AI governance is only a model-risk issue?
• What do security teams get wrong when they assume an AI app and an AI model have the same risk profile?