Security teams should evaluate a foundation model on more than performance metrics. Review the training data sources, cutoff date, alignment methods, model card disclosures, and safety testing history. Then test how the model behaves under adversarial prompting, jailbreak attempts, and prompt injection. A model that is accurate but poorly aligned can still create security, compliance, and misuse risk in production.
What to check before a foundation model enters your AI stack
Foundation model evaluation should start with provenance and governance, not benchmark scores. Training data sources, cutoff date, safety tuning approach, and disclosure quality all shape how much trust you can place in the model once it is exposed to real users, tools, and data. A strong model card is useful only if it is specific enough to support a real deployment decision.
That means looking for the evidence that explains model behaviour under pressure: what data it likely learned from, what post-training alignment was done, what red-team or safety testing was performed, and what known limitations remain. The point is to determine whether the model is well-characterised enough for the intended risk level, not whether it is simply impressive in a demo. For a broader GenAI governance lens, NIST’s NIST AI 600-1 GenAI Profile is a useful anchor because it connects pre-deployment evaluation to governance, provenance, and risk controls.
When the model will power a product that handles customer data, internal workflows, or automated decisions, that provenance review should also include dependency checks around hosting, update cadence, and vendor change control. If the provider cannot explain how the model was trained, tuned, and updated, you are accepting unknown behavior into production, which is a security decision as much as a product one.
How to test for adversarial behavior, not just capability
Security teams should evaluate how the model responds when the input is actively trying to steer it off course. Adversarial prompting, jailbreak attempts, prompt injection, and instruction collisions are not edge cases in generative AI, they are normal abuse paths once the model is exposed to untrusted content or tool-using workflows. The question is whether the model can be manipulated into leaking data, ignoring policy, or taking unsafe actions.
That testing should be scenario-based. Check whether the model resists attempts to reveal hidden instructions, whether it follows malicious content embedded in retrieved documents, and whether it preserves safe behavior when the prompt asks it to override policy or escalate privileges through a downstream tool. If the model can be tricked into unsafe outputs in a controlled test, it will usually fail faster in a live workflow where users are less disciplined and inputs are messier.
For teams building agentic or tool-connected applications, the right comparison point is not just generic application testing but attacker-driven abuse of model interaction boundaries. OWASP Agentic Applications Top 10 is relevant here because it highlights prompt injection, tool misuse, and privilege abuse as first-order design risks. When the model can influence actions, the safety test must include what happens when the model is persuaded to act outside the intended policy envelope.
Security teams should also pay attention to the gap between “model follows instructions” and “system remains safe.” A model may pass a chat safety test but still fail once it is wrapped in retrieval, tools, memory, or workflow automation. That is why the adversarial test plan needs to cover the full application path, not only isolated prompts.
What makes a model risky enough for production use
A foundation model becomes materially riskier when its behavior is under-documented, poorly bounded, or hard to monitor after deployment. The main danger is not only bad answers, it is confident bad behavior that is difficult to detect, attribute, or contain once the model is embedded in a business process. In practice, that can create compliance exposure, user harm, policy violations, or unsafe automation even when the model looks strong in evaluation.
Risk also rises when teams treat a general-purpose model as if it were already approved for a specific workload. A model that is acceptable for internal drafting may be unacceptable for regulated output, customer-facing recommendations, or tasks that can trigger actions in other systems. The deployment decision should reflect the worst credible misuse path, not the average expected case.
When teams want a governance baseline for model selection and ongoing oversight, the NIST AI 600-1 GenAI Profile aligns well with that need because it reinforces provenance, evaluation, and lifecycle controls rather than treating model quality as a one-time benchmark outcome. The practical test is whether you can explain why this model is acceptable for this use case, with evidence, before it touches production data or decisions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack surface, NIST AI 600-1 and NIST AI RMF set the technical controls, and ISO/IEC 42001:2023 define the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI 600-1 | GenAI Profile — Generative AI Profile | Covers GenAI governance, provenance, pre-deployment testing, and risk management. |
| Recommendation — Apply the GenAI profile to assess provenance, safety testing, and deployment risk before release. | ||
| NIST AI RMF | GOVERN — Govern | AI governance decisions must define acceptable risk, accountability, and oversight for model use. |
| MAP — Map | Model context, intended use, and stakeholder impact drive the risk profile and evaluation depth. | |
| MEASURE — Measure | Adversarial testing and safety evaluation quantify model behavior under misuse and attack. | |
| Recommendation — Establish governance for model approval, ownership, and risk acceptance before deployment. Map the model’s intended use, data flows, and stakeholder impacts before selecting it. Measure robustness with adversarial prompts, jailbreaks, and prompt-injection tests. | ||
| ISO/IEC 42001:2023 | A.6 — AI system lifecycle | AI lifecycle controls support pre-deployment evaluation, change control, and release decisions. |
| A.5 — Leadership and planning | Leadership and planning are needed to set risk criteria for acceptable model use. | |
| Recommendation — Embed model evaluation and change control into the AI system lifecycle. Define approval criteria and accountability for foundation model adoption. | ||
| OWASP Agentic AI Top 10 | A1 — Prompt Injection | Prompt injection is a primary adversarial test for GenAI applications using foundation models. |
| A3 — Tool Misuse | Foundation models can drive unsafe actions when connected to tools or workflows. | |
| Recommendation — Test the model and surrounding app for prompt-injection resistance before production. Constrain tool access so model outputs cannot trigger unsafe actions. | ||
| MITRE ATLAS | ATLAS — Adversarial AI Techniques | Provides adversarial AI techniques for red-teaming prompt abuse and model manipulation. |
| Recommendation — Use ATLAS techniques to structure red-team tests against model abuse paths. | ||
Practitioner Guidance
What to prioritise: Start with the questions that change deployment risk fastest, namely what data the model saw, what safety tuning it received, and how it behaves when an attacker tries to steer it. Those three checks tell you far more about operational suitability than headline capability scores.
What to verify: Require evidence you can audit later, including model card detail, evaluation notes, red-team results, and known failure modes. If the vendor cannot describe the model’s training cutoff, safety methods, or update process with enough precision to support a release decision, treat that as a gating issue rather than a documentation gap.
Decision rule: If the model will influence tools, content, or decisions that matter, only accept it when adversarial testing shows bounded behavior under prompt injection and jailbreak attempts. A model that is “mostly safe” in casual testing is not enough if a single failure can create user harm, policy breach, or unauthorised downstream action.
Practitioner takeaway: The right question is not whether the model is powerful, it is whether its behavior is sufficiently understood, constrained, and testable for the exact workflow you plan to trust it with.
Related resources from NHI Mgmt Group
- What should security teams evaluate before using compound AI systems in production?
- What do security teams get wrong about using generative AI for static application security testing?
- How should security teams evaluate asset-backed digital tokens before using them in a trading or payments model?
- How should security and AI teams evaluate model and prompt combinations before moving them into production?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 19, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
