Pre-release evaluation is the process of testing a model before it is made public or connected to operational systems. It typically combines benchmarking, red teaming, and risk review so that release decisions are based on evidence rather than vendor assurance alone.
Expanded Definition
Pre-release evaluation is the control point where an AI model, agent, or other automated decision component is assessed before it is exposed to users, tools, or production workflows. In NHI security, the term matters because release decisions can determine whether a model is allowed to invoke secrets, reach privileged APIs, or act through an agentic runtime. The practice usually combines benchmark testing, adversarial probing, and governance review, but definitions vary across vendors because no single standard governs this yet. For that reason, teams often map evaluation to broader risk disciplines such as the NIST Cybersecurity Framework 2.0 rather than treating it as a single test event.
At NHI Management Group, pre-release evaluation is best understood as a decision gate, not a scorecard. A model can pass one benchmark and still be unsafe if it can exfiltrate secrets, chain tools, or bypass intended privilege boundaries. The most common misapplication is treating pre-release evaluation as a one-time vendor acceptance check, which occurs when organisations approve a model before validating its tool access, data exposure, and rollback path.
Examples and Use Cases
Implementing pre-release evaluation rigorously often introduces delivery delay and test overhead, requiring organisations to weigh launch speed against the cost of shipping an unsafe model or agent.
- A finance team red-teams an assistant before enabling it to query payment systems, then blocks any prompt path that could reveal API keys or trigger unauthorised transactions.
- A platform team evaluates an internal coding agent against policy tests before granting it access to CI/CD runners and service account tokens.
- A security team reviews whether an AI workflow can read from a secrets manager or pass credentials through tool calls before production rollout, using lessons from the Ultimate Guide to NHIs.
- An enterprise gates release on both functional quality and misuse testing, aligning the process with NIST Cybersecurity Framework 2.0 functions such as Identify and Protect.
- A procurement team requires evidence that the model cannot escalate privileges across connected tools before it is approved for business use.
In practice, pre-release evaluation is also used to compare model variants, document residual risk, and decide whether a restricted launch is safer than full enablement.
Why It Matters in NHI Security
Pre-release evaluation reduces the chance that an AI system enters production with hidden pathways to sensitive identities, tokens, or machine-to-machine access. That matters because once an agent can call tools, the security problem shifts from model quality to operational authority. NHIMG research shows that 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools, which means an inadequately tested model may encounter credentials far earlier than teams expect. The same research also reports that 80% of identity breaches involved compromised non-human identities such as service accounts and API keys, underscoring how quickly AI misuse can become an identity incident. Pre-release evaluation is therefore a practical NHI control, not just an AI quality measure.
It is especially important when organisations connect a model to service accounts, approval workflows, or privileged automation without a mature rollback plan. The release decision should reflect not only what the model can do in theory, but also what it can reach in the live environment. Organisations typically encounter pre-release evaluation as an operational necessity only after a model has leaked a secret, escalated privileges, or triggered an unintended action, at which point the term becomes operationally unavoidable to address.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0, NIST Zero Trust (SP 800-207) and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | Agentic systems need testing before tool access and autonomous action are enabled. | |
| OWASP Non-Human Identity Top 10 | NHI-02 | Pre-release review should verify secrets, tokens, and service account exposure. |
| NIST CSF 2.0 | ID.RA-5 | Risk assessment informs release decisions when new AI capabilities are introduced. |
| NIST Zero Trust (SP 800-207) | AC-6 | Least-privilege principles apply when evaluating what the model can access at release. |
| NIST AI RMF | MAP | Pre-release evaluation supports structured AI risk identification and measurement. |
Validate agent behavior, tool use, and escalation paths before granting production authority.
Related resources from NHI Mgmt Group
- When should organisations prioritise runtime protection over pre-release checks?
- Why do AI agents require continuous access evaluation?
- What is the difference between static access control and continuous access evaluation?
- What is the difference between pre-deployment scanning and runtime protection?
