How should security teams evaluate foundation model risk before using it in a generative AI application?

What to review before a foundation model reaches a production AI app

Security review should begin with what the model is, not just how well it scores. Training data sources, cutoff date, alignment approach, and the model card tell you what the model may know, what it may omit, and what constraints were built into it. That context matters because a model can be useful and still be a poor fit for regulated, high-trust, or user-facing workflows.

Teams should also check whether the provider has published meaningful safety testing history and whether the disclosures are specific enough to support a real go-live decision. A strong model card is not a marketing summary; it is evidence that the provider understands the model's limitations, intended use, and known failure modes.

For model selection, treat source transparency as a control objective. If the vendor cannot explain data provenance, update cadence, or the scope of alignment work, you should assume more unknowns in production behavior, incident response, and compliance review.

How to test behavior, not just benchmark scores

Benchmark performance is only one part of the decision. A model that looks strong on accuracy can still be fragile under adversarial prompting, jailbreak attempts, or prompt injection, especially when it is embedded in a workflow that can retrieve data, call tools, or draft user-facing output. Those tests show how the model behaves when an attacker, a careless user, or a hostile document tries to steer it.

The most useful testing pattern is to combine safety evaluation with realistic abuse cases. Use prompts that try to override policy, elicit disallowed content, manipulate system instructions, or cause the model to disclose hidden context. Then test the same model in the actual application path, because a model that seems safe in isolation may fail once retrieval, plugins, or orchestration are added.

Evaluation should also distinguish between harmless odd outputs and material failure. Occasional hallucination is expected in generative systems; the question is whether the model can produce unsafe, misleading, or policy-violating output at a rate or severity that changes the deployment decision.

Why alignment and disclosures change the deployment decision

Alignment methods, safety tuning, and red-team history matter because they shape the model's default behavior under uncertainty. If the model has weak refusal behavior, inconsistent instruction hierarchy, or poor boundaries around sensitive topics, it may increase operational risk even when the underlying base model is technically capable.

Those concerns become more serious when the application handles regulated data, internal knowledge, or customer interactions. In that setting, the model's behavior can create security, compliance, and misuse risk even without a direct compromise. For teams formalising the decision process, NHIMG's AI Risk Assessment Guide is a practical starting point for structuring model review, and the NIST AI 600-1 Generative AI Profile Guide is useful when you want the risk discussion anchored to named GenAI risk areas.

Disclosures also support accountability. If the provider will not state what safety testing was performed, what changed since release, or where the model is expected to fail, security teams should treat the model as higher uncertainty and demand compensating controls before adoption.

Risk and Threat Considerations

Foundation models introduce risk when their training provenance, alignment quality, or prompt resilience is too opaque for the application's trust level. The main threat is not only harmful output, but also manipulation of the model into bypassing policy, leaking sensitive context, or generating content that creates downstream security or compliance exposure.

Failure mechanism: Weak safety tuning, inadequate adversarial testing, or prompt injection exposure lets the model be steered away from intended boundaries once it is connected to real data or tools.

Impact: The application can produce unsafe advice, disclose protected information, amplify fraud or misuse, or create a false sense of reliability that weakens human review.

Practitioner Guidance

What to verify: Before approval, verify that the provider can explain training provenance, model cutoff date, alignment method, and the exact scope of safety testing. If those details are missing, require a stronger deployment gate rather than compensating with user training alone.

Decision rule: If the model will touch sensitive content, external users, or automated actions, require adversarial prompting and prompt injection tests against the actual application path, not only against a standalone demo. A model that passes offline benchmarks but fails workflow testing is not ready for production use.

Practitioner takeaway: Treat foundation model evaluation as a trust decision, not a scorecard. The model is acceptable only when its disclosures, alignment evidence, and abuse-case testing all support the level of risk the application will inherit.

Related resources from NHI Mgmt Group

• Generative AI Application
• Generative AI
• What should security teams evaluate before using compound AI systems in production?
• What do security teams get wrong about using generative AI for static application security testing?