The main failure is assuming the system will behave the same way every time. Generative AI is probabilistic, so the same input can produce different output depending on context, retrieval, or configuration. That makes fixed-rule testing incomplete and can leave unsafe behaviour undiscovered until the model is already embedded in business processes.
Why This Matters for Security Teams
Treating AI like traditional software creates a false sense of control. Conventional application security assumes predictable logic, fixed outputs, and test cases that remain stable after release. Generative AI systems do not behave that way. Their outputs can shift with prompt wording, retrieval content, model updates, temperature settings, and surrounding tool access. That means the security question is not only whether the code is hardened, but whether the system’s behaviour is bounded enough to trust in production.
This matters because failures often show up in operational use, not in static testing. A model can pass a baseline review and still produce unsafe advice, leak sensitive context through a prompt injection path, or make inconsistent decisions once it is connected to retrieval and downstream automation. Guidance from the NIST Cybersecurity Framework 2.0 is useful here because it pushes teams to think in terms of governance, risk, and continuous control validation rather than one-time signoff.
In practice, many security teams encounter AI failure only after the system has already been embedded in a workflow and business users have started relying on its outputs.
How It Works in Practice
Security teams need to test AI as a system of model, data, prompts, retrieval, and tools, not as a single application binary. The practical issue is that the attack surface changes at inference time. A model may be technically sound while the surrounding context, retrieved documents, or agent permissions create unsafe outcomes. That is why current guidance suggests control testing must include prompt handling, input sanitisation, output validation, access boundaries, and monitoring for model drift or behaviour changes.
For AI systems that use retrieval or agents, identity and privilege become part of the security model. If an AI assistant can call tools, access files, or trigger workflows, it should inherit only the minimum permissions needed for that task. That is where NHI governance becomes relevant: the system often relies on machine identities, API keys, service accounts, or tokens that must be managed with the same discipline used for other high-value credentials. In a well-designed environment, teams also define which outputs are advisory, which are auto-executed, and which require human approval.
- Validate prompts and retrieved content for injection patterns before they reach the model.
- Apply least privilege to tool access, secrets, and service identities used by the AI workflow.
- Log prompts, outputs, retrieval sources, and tool actions for review and incident response.
- Test for unsafe or inconsistent behaviour after model updates, data changes, or configuration changes.
MITRE ATLAS helps teams think about adversarial AI tactics such as data poisoning and inference-time manipulation, while OWASP guidance for agentic AI is useful when the system can act rather than just generate text. The key point is that the security boundary is wider than the model itself. These controls tend to break down when teams connect AI to live business systems without a formal approval gate for tool use because hidden privileges turn model mistakes into real-world actions.
Common Variations and Edge Cases
Tighter AI control often increases operational overhead, requiring organisations to balance safety against delivery speed and user experience. That tradeoff is especially visible when teams want deterministic behaviour from systems that are inherently probabilistic. Best practice is evolving, and there is no universal standard for exactly how much variability is acceptable across use cases. A customer support assistant may tolerate limited variation, while an AI system that drafts financial, legal, or security guidance needs far stronger validation and human review.
Edge cases also appear when retrieval quality is poor, when the model is updated by a provider without advance notice, or when the same workflow is deployed across different regions and regulatory environments. In those environments, a single test suite is not enough. Teams should segment use cases by risk, define explicit output quality thresholds, and decide whether the system is decision support, content generation, or autonomous execution. Where the AI touches regulated data, incident handling should also include prompt logs, retrieval sources, and downstream actions, not just traditional application logs.
For governance-heavy environments, the most useful question is not whether the AI is “secure like software,” but which parts of the stack need software controls, which need model risk controls, and which need identity and privilege controls. That separation is what keeps assessments honest.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATLAS and OWASP Agentic AI Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST AI RMF and NIST AI 600-1 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV.RM | AI risk must be governed continuously, not treated as a one-time app signoff. |
| NIST AI RMF | GOVERN | The failure is largely governance, accountability, and lifecycle risk management. |
| MITRE ATLAS | T1589 | Adversarial AI tactics include manipulating inputs and surrounding context. |
| OWASP Agentic AI Top 10 | A01 | Agentic AI introduces tool-use and action risks beyond normal application flaws. |
| NIST AI 600-1 | GenAI profile guidance addresses validation, monitoring, and output controls. |
Set AI risk owners, review triggers, and ongoing control checks across the system lifecycle.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on August 11, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
