How should security teams evaluate GenAI models before production?

Why This Matters for Security Teams

GenAI model evaluation is not a model-quality exercise alone. It is a production risk decision about whether the system can be pushed into unsafe behavior by prompt attacks, retrieval poisoning, or tool misuse after deployment. NIST’s NIST AI 600-1 GenAI Profile is useful here because it frames GenAI risk in operational terms, not just accuracy metrics. NHIMG’s DeepSeek breach coverage shows how exposed data and model-adjacent infrastructure can quickly turn into a security event when secrets, chat histories, or backend credentials are reachable.

The practical mistake is treating a single benchmark score, vendor red-team report, or demo environment result as proof of readiness. Security teams need evidence that the model can resist realistic abuse patterns under the same retrieval sources, permissions, and integrations it will face in production. That includes direct prompt injection, indirect instruction injection from retrieved content, and any workflow where the model can act on behalf of a user. In practice, many security teams encounter unsafe model behavior only after the first production workflow has already connected the model to sensitive data or privileged tools.

How It Works in Practice

Production evaluation should combine adversarial testing, access review, and runtime control validation. The question is not only “does the model answer correctly?” but “does the system preserve intent when an attacker tries to bend it?” Current guidance suggests using a test harness that exercises the full chain: prompts, retrieval, system instructions, tool calls, and output handling. That is where many failures appear.

A useful pre-production review usually includes:

• Direct prompt attacks that try to override policy, reveal system messages, or trigger disallowed actions.
• Indirect instruction injection placed inside retrieved documents, web pages, tickets, or knowledge base content.
• Tool-abuse scenarios where the model is induced to call APIs, exfiltrate data, or chain actions across systems.
• Data-handling checks for secret leakage, sensitive memorization, and unintended retention of prior context.
• Rollback criteria that define what failure looks like and when release is blocked.

For identity and access questions, the model should be evaluated as part of a broader control plane, not as a standalone chatbot. NHI governance becomes relevant when the model or its surrounding services use secrets, tokens, or API keys to retrieve data or invoke tools. NHIMG’s Ultimate Guide to NHIs — The NHI Market is a useful reference for treating those credentials as governed identities rather than incidental configuration. The evaluation should verify that least privilege, short-lived credentials, and explicit approval boundaries still hold when the model is under adversarial pressure.

Best practice is evolving toward intent-aware testing, where authorization is checked at runtime against the action the model is trying to take, not just against a pre-approved role. That aligns well with the NIST AI 600-1 GenAI Profile, which emphasizes risk management across the lifecycle. These controls tend to break down when the model is connected to broad internal search, legacy secrets, or high-privilege automation because the attack surface becomes too dynamic for static test cases alone.

Common Variations and Edge Cases

Tighter pre-production testing often increases rollout time and evaluation cost, so organisations need to balance coverage against deployment urgency. The right depth depends on whether the model is read-only, can call tools, or can influence real transactions. A low-risk summarization assistant does not need the same abuse testing as a customer-facing agent with API access and retrieval into sensitive systems.

There is no universal standard for this yet, but some edge cases are clear. Open-ended agents need stronger tool-abuse scenarios than closed-domain classifiers. Models with external retrieval need adversarial content injection tests. Systems handling regulated data should add secret exposure checks, access-path review, and red-team replay. In environments where outputs are automatically executed, security teams should treat failure as a control-plane issue, not a prompt-engineering issue, because a single unsafe instruction can become a real-world action.

For teams that want a broader governance lens, the current consensus is that model evaluation should be repeated whenever prompts, retrieval sources, tool permissions, or identity bindings change. That is especially important in environments where NHIs are already fragmented or where secrets are hard to centralize, because security drift tends to appear between approvals rather than during them.

Related resources from NHI Mgmt Group

• How should security teams validate downloaded models before using them in production?
• How should security teams evaluate LLM defenses in production?
• What should security teams evaluate before using compound AI systems in production?
• How should security teams evaluate AI agent trust before production use?