AI agent performance monitoring: what APM misses in practice

TL;DR: AI agent performance monitoring needs more than traditional APM because a slow agent may be spending time reasoning, backtracking, or failing tools, while cost can rise even as latency improves, according to Codenotary. Conventional service metrics miss the reasoning loop, so observability has to cover task execution, tool behaviour, and spend together.

Editorial analysis by NHI Mgmt Group, based on content published by Codenotary: “AI Agent Performance Monitoring with OpenTelemetry”.

By the numbers:

• The post cites 11.5 seconds at P99 as an example of agent latency that still leaves the root cause hidden.
• The post says a tool in the example failed 32% of the time, which changed the agent’s overall performance profile.
• Codenotary says the fleet-wide tool-call error rate was 32.98% across 96,759 tool calls in the window.

Key questions

Q: How should security teams monitor AI agents without relying on traditional APM alone?

A: Security and platform teams should monitor AI agents with spans that expose reasoning steps, tool calls, errors, and cost, not just request latency.

Q: Why do long chain-of-thought outputs create operational risk in production AI systems?

A: Long chain-of-thought outputs increase token usage, latency, and GPU cost, which can make user-facing systems slower and more expensive.

Q: What are the signs that an AI agent is stuck rather than just busy?

A: Repeated backtracks, many failed subgoals, and a high count of tool calls with poor success rates are the main signs.

Practitioner guidance

• Instrument thought, tool, and cost spans Emit spans for reasoning steps, tool calls, durations, status, and token cost so the agent’s internal loop is visible in one trace.
• Track reasoning depth and backtracks Use reasoning depth and backtrack count as primary indicators of agent thrash, especially when latency alone does not explain poor outcomes.
• Review high-error tools as governance signals Treat tools with elevated error rates as control problems, because the agent may be repeatedly relying on a failing dependency or a broken workflow.

Bottom line: Traditional APM does not explain AI agent slowdowns well because it cannot see reasoning depth, backtracks, or tool-level failure patterns.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →