Capability evals measure upside on hard tasks where the baseline is weak, such as new tool patterns, long context, or multimodal inputs. Regression gates protect workflows that must not break, such as policy lookup, write actions, and safety-sensitive tasks. Teams should use both, but only regression gates should block promotion when critical paths fall below the required bar.
How capability evals and regression gates answer different questions
Capability evals ask, “Can this agent do something new or harder?” They are usually used when you want to measure upside on tasks where today’s baseline is weak, such as novel tool use, long-context reasoning, or multimodal inputs. Regression gates ask, “Did we preserve the behaviours that already have to be reliable?” They are focused on workflows where a break is unacceptable.
The practical difference is that capability evals are exploratory and comparative, while regression gates are protective and threshold-based. A team can celebrate a capability gain without treating it as safe to ship, because the score may improve on one hard task while critical workflow behaviour quietly degrades elsewhere.
That distinction matters because agents are often judged on the wrong axis. A higher benchmark result can hide a more fragile approval path, a weaker policy lookup flow, or a tool interaction that still fails under edge cases the business depends on.
What each one is designed to catch
Capability evals are best when the question is whether the agent has acquired a new skill, handles a broader input space, or performs better than a previous model on tasks that were previously out of reach. They are useful for selecting between candidate systems, measuring progress over time, and discovering which behaviours emerge as the system gets stronger.
Regression gates are best when the question is whether the agent still does the right thing in the places where failure is costly. That usually includes policy-sensitive lookups, write actions, approval flows, safety-sensitive tool calls, and other paths where a broken response can create user harm, compliance exposure, or operational disruption.
Teams often need both because the same release can be simultaneously better and riskier. A stronger agent may solve more cases, but also become more willing to act, more likely to overcall a tool, or more inconsistent in edge conditions that were previously stable.
How to use both without mixing up their roles
Capability evals should inform development, model selection, and tuning decisions. They tell you where the system is growing in competence and where it still fails to generalise, which helps you decide what to improve next.
Regression gates should inform release decisions. They protect the workflows that must remain within a known acceptable band, so they are the right mechanism for blocking promotion when a critical path falls below the required bar.
That is why the two should not be collapsed into one score. If you combine “can do more” and “must not break” into a single pass/fail number, you lose the ability to see whether a release is genuinely safer or merely more capable in the abstract. AAI Security Evaluations Guide is useful here because it frames eval design as a repeatable discipline, not a one-off benchmark exercise.
Risk and Threat Considerations
The main risk is promotion bias: teams can overtrust a capability gain and underweight the possibility that a seemingly stronger agent has become less reliable on the exact paths that matter most. In agent systems, that can translate into unsafe tool use, broken approvals, or degraded handling of policy and write actions.
Failure mechanism: The evaluation set over-rewards hard-task performance while missing regressions in high-value workflows, so the release decision is driven by upside metrics instead of operational safety.
Impact: A model that looks better on capability can still create broken business outcomes after deployment, especially when the failing path is rare, stateful, or only exercised under real user pressure.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 addresses the attack and risk surface, while NIST AI RMF sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI02 — Tool Misuse | Agents are judged on whether tool use stays safe under release conditions. |
| ASI03 — Identity & Privilege Abuse | Promotion gates should catch agent behaviour that exceeds intended authority. | |
| ASI08 — Cascading Failures | A regression in one agent path can propagate into broader workflow failure. | |
| Recommendation — Gate releases on tool-use regressions that could break critical workflows. Block releases when agent actions exceed approved authority or roles. Test critical paths for failures that could cascade into downstream harm. | ||
| NIST AI RMF | Govern | The question is about deciding how to govern evaluation and release decisions for agents. |
| Recommendation — Define separate governance criteria for capability gains and release-blocking regressions. | ||
Practitioner Guidance
Decision rule: Use capability evals to compare models, but let regression gates decide whether a release can move forward. If a critical path fails, treat that as a stop condition even when the agent shows clear capability improvement elsewhere.
What to verify: Make sure your regression set reflects the workflows that would be painful to break, not just the tasks that are easiest to benchmark. For agents, that usually means policy checks, tool writes, escalation paths, and any action with durable side effects.
What practitioners underestimate: A capability win can be real without being releasable. The mature operating pattern is to measure upside and safety separately, then require the safety gate to hold before you benefit from the capability gain.
Practitioner takeaway: Treat capability evals as evidence of potential and regression gates as evidence of readiness, because only the latter should have veto power over promotion when the workflow is mission-critical.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 31, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org