Notifications
Clear all
Topic starter
20/08/2026 5:34 pm
TL;DR: An AI agent can improve from 53.8% to 83.1% decision alignment by mining human overrides as a context graph, not by retraining or changing source code, according to Arize. The governance lesson is that agent control depends on capturing disagreement, preserving reviewer rationale, and continuously recalibrating permissions as business context shifts.
NHIMG editorial — based on content published by Arize: Building a self-improving agent on a context graph of human disagreement
By the numbers:
- 33% of organisations report their AI agents have accessed inappropriate or sensitive data beyond their intended scope.
Questions worth separating out
Q: How should organisations turn AI governance policy into enforceable controls?
A: Organisations should translate policy into specific approval gates, data access rules, logging requirements, and change controls that sit inside the AI lifecycle.
Q: Why do AI agents need human review trails if the model is already accurate?
A: Accuracy alone does not solve governance when business context changes faster than model behaviour.
Q: What do security teams get wrong about governing AI agents?
A: They often treat agents like another automation layer instead of governed non-human actors with their own access paths.
Practitioner guidance
- Capture overrides as structured evidence Record the original agent decision, reviewer rationale, and exception tag in a machine-readable schema so the same pattern can be reused in future policy evaluation.
- Separate policy updates from model updates Route fast-changing business rules into runtime configuration or policy documents instead of changing model weights when the issue is governance, not model capability.
- Define bounded agent discretion Set explicit thresholds for when the agent may approve, reject, or escalate, and make the escalation path visible in the same trace session as the decision.
What's in the full article
Arize's full post covers the operational detail this post intentionally leaves for the source:
- The full procurement-agent demo structure, including the exact trace layout used to capture overrides and precedent tags.
- The cycle-by-cycle tuning logic that turned review comments into runtime configuration changes.
- The implementation pattern for mining trace sessions with a Claude Agent SDK tool.
- The GitHub-based demo workflow for reproducing the context-graph loop end to end.
👉 Read Arize’s analysis of building a self-improving agent on a context graph →
Human overrides and AI agents: what context graphs change?
Explore further
20/08/2026 6:20 pm
Human override trails are becoming the most valuable control artifact in agent governance. The article shows that the real policy often lives outside the policy document, in review comments and exception handling. That makes override data a governance asset, not a by-product, because it captures the decision logic that actually governs production behaviour. Practitioners should treat override capture as part of the control design, not as a reporting afterthought.
A question worth separating out:
Q: How do IAM and PAM teams split responsibility for AI agent access?
A: IAM should define what the agent can reach, while PAM should control when elevated access is available and how it is revoked. For AI agents, those responsibilities must be coordinated because programmatic identities do not fit a human session model. If scope and elevation are managed separately without a shared lifecycle view, privilege can persist longer than anyone expects.
👉 Read our full editorial: Human overrides as a context graph for better AI agent control
