Model evaluation tells you whether a model responds safely in isolation. Contract-based governance tells you whether the agent can actually stay within approved tools, data, memory, and reasoning limits while interacting with real systems. That distinction matters because most operational risk appears when the agent acts, delegates, or persists context across sessions.
Why This Matters for Security Teams
Model evaluation is useful, but it only measures the model in a controlled test context. AI agents introduce a different risk surface because they can call tools, retrieve data, write to systems, and keep state across turns or sessions. That means the real question is not just whether the model answered safely, but whether the agent stayed inside approved execution boundaries. Guidance from the NIST AI Risk Management Framework aligns with this shift from output quality to lifecycle governance.
For security teams, the failure mode is usually not a dramatic model error. It is a small authority overreach, such as an unapproved API call, an overly broad retrieval source, or retained context that should have been discarded. Contract-based governance gives practitioners a way to define what the agent may access, what it may retain, what it may delegate, and when it must stop. That is especially important in environments where the agent can trigger downstream automation or influence business decisions.
In practice, many security teams encounter agent risk only after a tool call, data exposure, or workflow abuse has already occurred, rather than through intentional model testing.
How It Works in Practice
Contract-based governance turns vague expectations into enforceable runtime conditions. A contract can specify approved tools, allowed data classes, memory retention limits, escalation thresholds, human approval points, and required logging. Instead of trusting a model to “behave,” the organisation constrains the agent’s actions before they reach production systems. This aligns well with the control themes in OWASP Agentic AI Top 10 and the threat patterns catalogued in the MITRE ATLAS adversarial AI threat matrix.
A practical contract usually includes four layers:
- Identity and authorisation: what the agent is permitted to do and on whose behalf.
- Tool policy: which APIs, browsers, databases, and internal services it may invoke.
- Data governance: what content can be read, summarised, stored, or forwarded.
- Behavioural limits: when the agent must request approval, halt, or hand off to a human.
These controls are most effective when enforced outside the model, at the orchestration or policy layer. That way, a prompt injection, hallucinated instruction, or compromised retrieval source cannot directly expand the agent’s authority. Logging is also central, because audits need to show not only what the model said, but which contract rules were checked and whether the agent complied. This is where NHI thinking becomes relevant: if the agent can act on systems, it needs a governed identity posture, not just an evaluated prompt.
These controls tend to break down when agent tooling is assembled ad hoc across multiple plugins, because policy enforcement becomes inconsistent and hard to audit.
Common Variations and Edge Cases
Tighter contract-based governance often increases friction for teams that want fast experimentation, requiring organisations to balance safety against workflow speed. Best practice is evolving, and there is no universal standard for contract syntax, but the operating principle is stable: evaluation alone cannot govern action. For high-impact use cases, contracts should be stricter than for low-risk drafting or summarisation tasks.
Some environments need additional nuance. A customer support agent may need broad retrieval access but very limited write permissions. A software delivery agent may need repository access, but only through scoped credentials and approved pull requests. A finance or regulated workflow may require explicit human sign-off before any external transaction or record update. The more the agent can change state, the more important it becomes to bind its permissions to the minimum necessary scope.
Contract design also needs to account for persistence. If an agent carries memory across sessions, then stale context can become a governance problem even when the model itself is behaving. That is why runtime policy, session boundaries, and approval workflows matter as much as evaluation scores. For teams building or reviewing these systems, the CSA MAESTRO agentic AI threat modeling framework and the NIST AI Risk Management Framework both reinforce this governance-first approach.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, MITRE ATLAS and CSA MAESTRO address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | AI RMF centers govern, map, measure, and manage for agent risk. | |
| OWASP Agentic AI Top 10 | Agentic app risks include tool abuse, prompt injection, and unsafe autonomy. | |
| MITRE ATLAS | ATLAS captures adversarial techniques that target model and agent behaviour. | |
| CSA MAESTRO | MAESTRO is relevant for governing autonomous agent workflows and trust boundaries. | |
| NIST CSF 2.0 | PR.AC-4 | Agent tool access must be restricted to least privilege and approved scopes. |
Document trust boundaries and policy checks for each agent action in the orchestration layer.
Related resources from NHI Mgmt Group
- When do AI agents become an NHI governance problem instead of an automation tool?
- How do you know if your identity governance model is keeping up with AI agents?
- What breaks when AI agents use session-based micropayments without governance?
- When does MCP provide a better governance model than CLI for AI agents?
