Agent platforms make it easy to build and publish AI agents, and just as easy to end up with hundreds of them running under their makers' credentials, shared more widely than intended and calling tools nobody reviewed. This health check is a recurring review of an agent platform in production: low-code agent builders, agent hosting services and internal agent runtimes built on open frameworks. It gives thirteen metrics with formulas and thresholds for inventory, ownership, identity, sharing, tool permissions, logging and containment. It is for AI platform owners, security architects, identity teams and the people who sign off agent risk.
This page measures; it does not repeat the governance guidance. For how to configure environments, connector policies, sharing and identity on the platform, use the Low-Code Agent Platforms Guide, and for code-first runtimes the Securing Agent Frameworks Guide. The metrics are vendor-neutral and apply to any platform that hosts agents with tools and credentials.
What this health check covers
The review covers every agent the platform can run in production and pre-production, the identities and connections those agents use, the tools and connectors they can call, who they are shared with, the logs they produce and the controls to stop them. It does not cover model quality or evaluation, and it does not replace pre-production security testing of individual agents.
- Who runs it: the agent platform owner collects the data; the identity team validates the identity and credential metrics; security operations runs the containment drill; the head of AI governance or CISO signs off.
- Scope by environment: production is mandatory. Include developer and experimentation environments for inventory metrics (AGP-01, AGP-10), because agents with live connections often sit there.
- Typical questions it answers: "how do we audit our AI agents", "what agent governance KPIs should we report" and "how many agents run with personal credentials".
Key takeaways
- Inventory is the first metric. Reconcile what the platform's admin export shows with what your agent registry holds; every gap is an agent nobody governs.
- Count agents running as people. Agents using a maker's personal credentials inherit everything that person can do, and break or persist when that person leaves.
- Measure sharing and tool reach. Organisation-wide or anonymous sharing and unrestricted outbound tools turn a helpful agent into a data exfiltration path.
- High-impact actions need approval evidence. Measure the share of agents with write or payment tools that require a human approval step.
- Time the kill switch. How long it takes to stop one agent, and every agent on the platform, is the best test of readiness.
How to use this template
Copy the metrics table and the findings log into your own tracker, then run this Agent platform health check at the cadence set out below. Collect each figure from the data sources described, score it against the green, amber and red thresholds, and record every amber or red result as a finding with an owner and a due date. Keep each completed review so you can compare results quarter on quarter: the trend matters as much as any single score.
Cadence and sign-off
| Review | Frequency | Owner | Approver | Evidence kept |
|---|---|---|---|---|
| New agents and sharing changes | Weekly | Agent platform team | Agent platform owner | Admin export diff, new connectors, sharing changes |
| Ownership and leaver reconciliation | Monthly | Identity team | Agent platform owner | Agent owner list joined to HR leaver feed, actions taken |
| Full metrics review (this page) | Quarterly | Agent platform owner | CISO and head of AI governance | Metrics table, extracts, findings log |
| Containment drill | Quarterly | Security operations | Agent platform owner | Drill timeline, time to stop, residual activity found |
| Connector and tool policy review | Quarterly, and when a new connector is approved | AI governance team | Head of AI governance | Policy versions, approved connector list, exceptions |
Metrics
NHI Mgmt Group recommended thresholds; adjust to your risk appetite. "High-impact tool" means any tool or connector that can write, delete, send external messages, move money or change access.
| ID | Metric | Formula | Data source type | Green | Amber | Red |
|---|---|---|---|---|---|---|
| AGP-01 | Registry coverage | Agents in the platform admin export that are also in the agent registry ÷ agents in the platform admin export | Platform agent inventory export, agent registry | At least 98% | 90% to below 98% | Below 90% |
| AGP-02 | Agents with a current accountable owner | Production agents whose named owner is an active employee in the owning team ÷ production agents | Agent registry joined to HR worker list | 100% | 95% to below 100% | Below 95% |
| AGP-03 | Agents running on personal credentials | Production agents whose connections authenticate as the maker's personal account ÷ production agents with at least one connection | Platform connection inventory with authentication type | 0% | Above 0% to 5%, with exception | Above 5% |
| AGP-04 | Over-shared agents | Agents shared organisation-wide, externally or anonymously without an approval record ÷ total published agents | Platform sharing settings export, approval records | 0% | Above 0% to 2% | Above 2% |
| AGP-05 | Connector policy violations | Agents using a connector outside the approved list, or combining internal sensitive and external connectors against policy ÷ total agents | Connector usage export, connector policy | 0% | Above 0% to below 1%, with tickets | 1% or more |
| AGP-06 | Human approval on high-impact tools | Agents with at least one high-impact tool that require human approval before that tool runs ÷ agents with at least one high-impact tool | Agent tool configuration export | 100% | 90% to below 100% | Below 90% |
| AGP-07 | Unrestricted outbound reach | Production agents with a generic web request or code execution tool and no egress allow list ÷ production agents | Agent tool configuration, egress policy | 0% | Above 0% to 5%, sandboxed | Above 5% |
| AGP-08 | Stale agent credentials | Agent service identities, keys or tokens older than the rotation policy ÷ total agent credentials | Secrets store inventory, identity provider app credential list | Below 5% | 5% to 20% | Above 20% |
| AGP-09 | Secrets in agent configuration | Agents with credentials found in prompts, instructions or plain configuration fields ÷ agents scanned | Configuration export scanned for secret patterns | 0 | None in production; found in development | Any in production |
| AGP-10 | Dormant enabled agents | Enabled agents with no run in 90 days ÷ enabled agents | Platform run history | Below 5% | 5% to 15% | Above 15% |
| AGP-11 | Tool call logging coverage | Production agents whose tool calls, with agent identity and on-behalf-of user, reach the SIEM ÷ production agents | Platform audit log settings, SIEM source inventory | 100% | 90% to below 100% | Below 90% |
| AGP-12 | Pre-production security review | Production agents with a recorded security review or test before release ÷ agents released to production in the period | Change records, agent registry | 100% | 90% to below 100% | Below 90% |
| AGP-13 | Time to stop an agent | Median minutes from stop decision to the agent's last tool call, across drills in the period | Drill records, tool call logs | Under 15 minutes | 15 to 60 minutes | Over 60 minutes or no drill |
Report AGP-03, AGP-04 and AGP-06 per environment and per business unit as well as in total. A platform-wide green figure can hide one department where most agents run on personal accounts, and that is where remediation effort should go first.
How to collect the data
- Export every agent from the platform's admin inventory across all environments, with creator, owner field, environment, status, sharing scope, connectors and tools, and last run date.
- Reconcile with the registry. Join the export to your agent registry on a stable agent identifier. Agents in the export but not in the registry give AGP-01; registry entries with no agent are stale records to close. The AI agent registry guide sets out the fields.
- Join owners to HR data. Match each owner to the HR worker list for AGP-02. Feed leavers into the Joiner-Mover-Leaver Guide process.
- Classify connections by authentication type: end-user delegated, dedicated service identity or maker's personal account. Only the last counts against AGP-03.
- Classify tools. Tag each tool and connector as read-only or high-impact once, keep the list versioned, then compute AGP-06 and AGP-07 from the agent tool configuration export.
- Scan configuration for secrets. Run a secret pattern scan over exported instructions, prompts and configuration fields for AGP-09.
- Check credential age from the secrets store and the identity provider's app credential list for AGP-08.
- Confirm logging end to end. Check audit settings per environment, then confirm the SIEM received tool call events from each production agent in the last seven days.
- Run the containment drill. Disable a test agent and, separately, revoke its credentials, and time both until tool calls stop. The AI Agent Kill Switch and Containment Guide describes the layers to test.
Red flags that need action now
- A production agent with access to finance, HR or customer data running under a maker's personal account.
- An agent shared anonymously or externally that can read internal data.
- An agent that can send email, make payments or change access with no human approval step.
- Credentials in an agent's instructions or prompt text.
- An agent whose owner has left and which still runs on a schedule.
- Tool calls that do not appear in the SIEM, or appear without the agent identity.
- No one can say how to stop every agent on the platform at once.
Where a red flag suggests misuse, for example unexpected external sends or data volumes, switch to the AI Agent Incident Response Playbook before cleaning up, so tool call logs and agent configuration are preserved.
Findings log
| ID | Finding | Metric | Severity | Owner | Due date | Status |
|---|---|---|---|---|---|---|
| HC-AGP-01 | [Description, population and figure] | AGP-01 | [Critical / High / Medium / Low] | [Role] | [Date] | [Open / In progress / Closed / Risk accepted] |
| HC-AGP-02 | [Description] | AGP-03 | [Severity] | [Role] | [Date] | [Status] |
| HC-AGP-03 | [Description] | [Metric ID] | [Severity] | [Role] | [Date] | [Status] |
Suggested due dates: red flags within 7 days, red metrics within 30 days, amber metrics within 90 days. Risk acceptance needs the agent platform owner's signature and an expiry date.
Remediation map
| Finding type | Where to fix it |
|---|---|
| Inventory, ownership and dormant agents (AGP-01, AGP-02, AGP-10) | Low-Code Agent Platforms Guide (lifecycle) and AI agent registry |
| Personal credentials and stale or exposed secrets (AGP-03, AGP-08, AGP-09) | Agentic AI Identity Guide and Runtime Secrets Injection for AI Agents Without Prompt Exposure |
| Sharing and connector policy (AGP-04, AGP-05) | Low-Code Agent Platforms Guide (platform configuration) |
| High-impact tools without approval (AGP-06) | AI Agent Action Approvals and Spending Limits Guide and OWASP LLM Top 10 LLM06 Excessive Agency Guide |
| Unrestricted outbound reach (AGP-07) | Agent Egress Control and Network Isolation Guide |
| Logging gaps (AGP-11) | AI Agent Observability and Incident Response Guide |
| Releases without review (AGP-12) | AI Agent Security Testing Checklist |
| Slow containment (AGP-13) | AI Agent Kill Switch and Containment Guide |
Related guides
- Low-Code Agent Platforms Guide: the governance controls this review measures.
- Securing Agent Frameworks Guide: the equivalent controls for code-first agent runtimes.
- MCP, Protocols & Agent Platforms Maturity Model: where agent platform governance sits on a maturity scale.
- AI Agent Incident Response Playbook: what to do when a finding looks like misuse.
- AI gateway health check: the companion review for model traffic the agents generate.
References
- OWASP: Top 10 for LLM Applications 2025 (LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM06 Excessive Agency)
- OWASP: Top 10 for Agentic Applications for 2026
- NIST: AI Risk Management Framework
- NIST: SP 800-53 Rev. 5 (AC-2 Account Management, AC-6 Least Privilege, IA-5 Authenticator Management, IA-9 Service Identification and Authentication, AU-12 Audit Record Generation, CM-8 System Component Inventory)
- NIST: Cybersecurity Framework 2.0 (ID.AM-02 software and service inventories, PR.AA-05 access permissions, DE.CM-09 monitoring of computing software and runtime environments)
- CIS: Critical Security Controls v8.1 (Control 2 Inventory and Control of Software Assets, Control 5 Account Management, Control 8 Audit Log Management)
Related NHI Mgmt Group resources: Agentic AI Identity Guide, Agent Permission Drift, Agent Controls & Monitoring Maturity Model.