Agentic AI governance is shifting from model control to system control

At a glance

What this is: Holistic AI’s account of the Great Agent Hack 2025 says production-ready agentic AI depends on observability, red-teaming, and multi-agent governance rather than isolated model oversight.

Why it matters: For IAM and security teams, this matters because agentic systems behave like governed runtime actors, so identity, tool access, auditability, and control boundaries must be designed together.

By the numbers:

• The Great Agent Hack 2025 brought together more than 200 builders and produced 51 submissions across three challenge tracks.

👉 Read Holistic AI's analysis of the Great Agent Hack 2025 and agentic AI governance

Context

Agentic AI governance is the practical problem this article points to: when systems plan, call tools, coordinate across roles, and change behaviour over time, classic model-level oversight is not enough. The primary issue is not whether a model can answer a prompt, but whether the full agent workflow can be observed, constrained, and audited across data, model, and infrastructure layers.

That matters to identity teams because every agent that can act, delegate, or access tools needs a clear trust boundary, lifecycle, and entitlement model. In that sense, the article sits at the intersection of AI governance and non-human identity management, where runtime behaviour, access patterns, and accountability must be managed together rather than as separate controls.

The hackathon setting is atypical in scale but typical in direction: it reflects where enterprise AI programmes are heading once prototypes become operational systems.

Key questions

Q: How should security teams govern agentic AI that can execute IAM tasks?

A: Start by treating the agent as an NHI with bounded authority, explicit ownership, and revocation procedures. Require human approval for high-risk actions, log every decision path, and enforce least privilege at the workflow level. If the agent cannot be audited or rolled back, it is not yet ready for autonomous IAM execution.

Q: Why do AI agents complicate existing IAM and authorization models?

A: AI agents complicate IAM because they turn natural language into execution, which can cross systems faster than human review can intervene. Traditional standing access models assume stable actors and predictable workflows. Agents are more dynamic, so the control point must shift to ephemeral authorization, contextual claims, and continuous validation.

Q: How do security teams know if agent observability is actually working?

A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session. If those signals are disconnected, the logs may show traffic but not governance. The test is whether investigators can reconstruct the path of a meaningful agent action from start to finish.

Q: Who is accountable when an AI agent causes a security incident?

A: Accountability should sit with the business owner, the system owner, and the security function together, because agent behaviour crosses operational boundaries. Organisations need a defined owner for approval, monitoring, and retirement, plus audit evidence that shows what the agent accessed and why.

Technical breakdown

Why observability becomes the control plane for agentic AI

AI observability extends logging, metrics, and traces into the full agent stack, including prompts, outputs, tool calls, latency, costs, and security events. For agentic systems, this is not optional telemetry. It is the only practical way to reconstruct why an agent acted, whether its behaviour drifted, and where a decision chain crossed a governance boundary. Without that evidence, teams cannot separate model error from workflow error or tool misuse.

Practical implication: instrument agent workflows with trace-level logging, policy checkpoints, and immutable audit trails before expanding production access.

How multi-agent systems change governance and privilege control

Multi-agent designs distribute work across planners, reviewers, executors, and verifiers. That architecture improves flexibility, but it also creates more identities, more tool access paths, and more opportunities for failure through interaction rather than isolated model output. Governance therefore shifts from supervising one decision-maker to governing a chain of delegated actions. In identity terms, each agent role becomes a non-human identity with its own scope, boundary, and accountability requirement.

Practical implication: assign least-privilege access per agent role and review delegation paths as rigorously as human access chains.

Why red-teaming must test behaviour, not just prompts

Systematic red-teaming for agentic AI goes beyond prompt injection probes. It needs to test tool misuse, semantic manipulation, coordinated failure across agents, and the way behaviour changes under pressure. The article’s emphasis on structured evaluation reflects a broader shift in AI security: weaknesses often emerge only when the system is exercised end to end, with external tools, memory, and workflow state all in play. That makes continuous adversarial testing part of operational governance, not a one-time assurance activity.

Practical implication: build recurring adversarial tests around tool access, delegation logic, and state persistence, then feed findings into change control.

NHI Mgmt Group analysis

Observability is becoming the first governance requirement for agentic AI. The article shows that teams are converging on traces, dashboards, and step-level inspection because agent behaviour cannot be governed after the fact. When prompts, outputs, tool use, and system events are visible together, teams can separate acceptable variability from policy failure. The practical conclusion is clear: without full-stack observability, agent governance is speculative.

Multi-agent systems create a non-human identity management problem, not just an AI design problem. Each agent role behaves like a runtime actor with delegated responsibility, which means entitlement scope, tool boundaries, and lifecycle control matter as much as model quality. This is where identity governance and AI governance meet. The field should treat agent roles as governed non-human identities, not as invisible implementation details.

Red-teaming is shifting from model evaluation to system resilience engineering. The hackathon’s focus on structured testing and behavioural stress reveals that agent security failures often arise from interaction effects, not single prompt defects. That aligns with NIST AI RMF and the OWASP Agentic AI Top 10, where operational assurance depends on repeated testing across the whole workflow. Practitioners should expect continuous evaluation to become a baseline control, not a specialist exercise.

Agent governance debt: when teams deploy autonomous or semi-autonomous workflows before defining traceability, delegated access, and rollback rules, they create hidden control debt that compounds over time. This debt shows up as unclear accountability, weak audit evidence, and overreliance on manual intervention when something goes wrong. The longer it remains unaddressed, the harder it becomes to prove safe operation at scale. Practitioners should treat governance design as part of delivery, not post-deployment cleanup.

Agentic AI is pushing security teams toward control design that spans AI, identity, and infrastructure. The article makes clear that model performance alone does not create enterprise readiness. Success depends on how access, observability, and response are wired into the system from the start. For practitioners, the message is to build cross-functional controls before production rollout, not after the first failure.

What this signals

Agent governance debt: the longer teams defer traceability, delegated-access rules, and rollback design, the harder it becomes to prove control once agentic workflows reach production. That is why identity governance must move upstream into architecture decisions, not remain a post-deployment review activity.

For programmes that already manage NHI estates, the practical shift is to treat agent roles as lifecycle-managed subjects with named owners, scoped access, and offboarding triggers. The governance model is converging with workload identity management, which means existing IAM and PAM disciplines need to extend into AI runtime design.

Teams should align control design with NIST AI Risk Management Framework and the OWASP Top 10 for Agentic Applications 2026 where agent behaviour, tool access, and accountability intersect. The immediate programme signal is simple: if you cannot explain an agent action, you do not yet control the agent.

For practitioners

• Define agent identities and privilege boundaries Treat each planner, executor, reviewer, and verifier as a distinct non-human identity with a scoped entitlement set, named owner, and explicit lifecycle. Link access to task class rather than generic system access so agent roles cannot drift into unrestricted tool use.
• Instrument end-to-end traces for every agent workflow Capture prompts, tool calls, outputs, latency, errors, and policy decisions in a single traceable record so investigators can reconstruct behaviour across the full chain. Correlate those traces with change events and policy updates to spot drift early.
• Run recurring adversarial tests on delegation logic Test how agents behave when instructions conflict, tools return unexpected results, or one agent tries to influence another. Focus on delegation paths, memory persistence, and tool misuse rather than only prompt injection.
• Establish rollback and containment rules before production Predefine when an agent loses tool access, when a workflow is paused, and how human review is triggered after anomalous behaviour. Make the containment path part of the deployment design, not an incident-only decision.

Key takeaways

• The article’s core message is that agentic AI readiness depends on observability, red-teaming, and delegation control, not model performance alone.
• The governance shift is from supervising a single system to managing multiple runtime actors, each with its own access path and accountability burden.
• Enterprises should treat agent identities as lifecycle-managed subjects and build traceability before they expand production access.

Key terms

• Agentic AI: Autonomous AI systems capable of planning, deciding, and taking actions — including calling APIs, writing code, and orchestrating other agents — with minimal human oversight. Agentic AI introduces new NHI risks as agents must authenticate to external services.
• AI observability: AI observability is the ability to see how AI systems are being used, what information they process, and what actions they trigger. In security programmes, it extends beyond uptime or model quality to runtime visibility, policy enforcement, and audit evidence across human and agent-driven use cases.
• Delegated tool privilege: Delegated tool privilege is the effective authority an AI agent inherits from the systems it can call, even when it is not the owner of those systems. It is the practical blast radius created by connectors, credentials, and runtime permissions. This is often the real control boundary in agentic environments.
• Agent Governance: Agent governance is the set of policies, controls, and evidence required to manage autonomous software as a non-human identity. It covers consent, tool access, lifecycle review, audit logging, and revocation so that an agent remains bounded as its workflows change.

What's in the full article

Holistic AI's full blog covers the operational detail this post intentionally leaves for the source:

• Track-by-track descriptions of the winning agent designs and the specific techniques each team used to score well
• Judge and sponsor commentary on why observability, safety, and performance were evaluated together
• Examples of the red-teaming and behavioural profiling work that led to the grand champion result
• The collaborative research follow-on that turns hackathon prototypes into broader evaluation work

👉 Holistic AI's full blog covers the winning projects, evaluation themes, and research collaborations in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It is designed for practitioners building the control foundations that AI-driven systems now depend on.