Capability evals are designed to uncover what an agent still cannot do, so they use harder tasks with low initial pass rates. Regression evals protect already accepted behavior, so they keep near-perfect tasks in the suite after changes. In practice, teams use capability tests to find gaps and regression tests to prevent backsliding.
Capability evals ask “what can this agent do?”
Capability evals are designed to discover the ceiling of an AI agent’s performance, not to certify that it is safe after deployment. That means they should include tasks the agent is likely to fail at first, so the suite reveals missing reasoning, weak tool use, brittle planning, or poor instruction following. In agentic systems, the useful question is whether the benchmark is stretching the model’s current limits, not whether it preserves a stable pass rate.
That framing matters because a capability suite is most valuable when it is diagnostic. If the agent can solve everything already, the eval is not telling you where the edge is. Practitioners often use harder tasks, broader scenarios, or lower initial pass-rate targets so the results expose gaps that are otherwise hidden by easy benchmarks.
When teams interpret capability evals well, they treat them as discovery instruments. A failure is not just a score, it is a signal about what the agent still lacks, whether that gap is in planning, tool selection, context handling, or recovery from partial success.
Regression evals ask “did we break what already worked?”
Regression evals serve a different purpose: they protect known-good behavior after prompts, tools, models, policies, or orchestration logic change. The suite should contain tasks the agent already passes reliably, because the goal is to catch backsliding, not to discover new weaknesses. In practice, that means keeping a stable set of near-perfect scenarios in rotation and comparing new runs against the accepted baseline.
This distinction is especially important for agent systems because changes are often local but the effects are systemic. A model upgrade, a tool wrapper change, or a modified safety policy can alter behavior in ways that are not obvious from the code diff alone. Regression evals are the mechanism that tells you whether previously acceptable behavior is still intact.
Teams should expect regression suites to be conservative. If the pass rate drops on an established scenario, that is usually a release-quality problem, not a sign that the benchmark became more interesting.
What changes in practice, and why teams need both
Capability evals and regression evals answer different operational questions, so they should not be mixed into a single undifferentiated score. Capability evals help with roadmap decisions, model selection, and identifying where an agent needs additional scaffolding. Regression evals help with release gating, change control, and confidence that an update did not erode established behavior.
The best way to use them together is to separate the threshold logic. Capability results should tell you what to improve next. Regression results should tell you whether a change is acceptable to ship. If a team uses only capability testing, it can celebrate progress while missing subtle degradation. If it uses only regression testing, it can preserve yesterday’s behavior while never learning where the agent is still weak.
For a practical example of why change control matters in agentic systems, compare this distinction with observed real-world agent misuse and overreach in AI Agents: The New Attack Surface report and with compromise patterns in LLMjacking: How Attackers Hijack AI Using Compromised NHIs. The underlying lesson is that agent behavior needs both discovery of gaps and protection against drift.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | A1 — Agent Goal Hijacking and Prompt Injection | Agent evals must catch failures in instruction following and tool use. |
| Recommendation — Design evals that reveal goal hijacking and prompt-injection weaknesses before deployment. | ||
| NIST AI RMF | GOV — Govern | Capability and regression testing are part of AI governance and lifecycle control. |
| Recommendation — Set governance gates that separate exploratory capability testing from release regression checks. | ||
| CSA MAESTRO | L1 — Agent Lifecycle Security | Agent behavior must be validated across updates to preserve safe operation. |
| Recommendation — Validate agent changes against stable baseline behaviors before promoting them. | ||
| NIST CSF 2.0 | PR.IP — Information Protection Processes and Procedures | Regression evals support controlled changes and repeatable assurance practices. |
| Recommendation — Maintain repeatable test procedures that detect behavior drift after changes. | ||
Practitioner Guidance
What to prioritise: Keep capability and regression suites separate in both naming and decision logic. If a test is meant to measure improvement potential, let it fail when the agent is still immature; if a test is meant to protect shipped behavior, treat any meaningful drop as a release risk.
What to verify: Make sure every regression case is anchored to an accepted baseline and every capability case has an explicit purpose, such as tool use, multi-step reasoning, or recovery from partial failure. A mixed suite is hard to interpret because a single pass rate then hides two different questions.
Common mistake: Teams often promote a capability benchmark into a regression gate without rebalancing the difficulty. That creates false alarms and encourages tuning to the test instead of preserving real operational behavior.
Practitioner takeaway: Use capability evals to expose the frontier and regression evals to freeze the floor, because mature agent programs need both exploration and change control.
Related resources from NHI Mgmt Group
- What is the difference between capability evals and regression gates for agents?
- What is the difference between using AI to assist ethical hacking and giving autonomous agents full hacking capability?
- What is the difference between controlling AI agents by credential and controlling them by capability?
- What is the difference between managed identities and hardcoded secrets for AI agents?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 20, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
