Why does keyword search miss important agent failures?

Why keyword search misses agent failures

Keyword search is useful for finding obvious incidents, but it is a weak way to detect agent failure modes. The failure can be in the decision path, not the wording: a model may select the wrong tool, inherit the wrong context, or continue after a policy boundary has already been crossed. That means the signal lives in traces, actions, and outcomes.

For agent systems, that distinction matters because the same request can look “normal” in text while still producing the wrong behaviour. A search over prompts or logs that only keys on suspicious terms will miss silent failures, partial failures, and failures that only become visible after an external side effect, such as a bad API call or an unauthorised workflow step.

Practical evaluation starts by treating the agent trace as the unit of analysis. The important question is not whether the agent said the expected thing, but whether it completed the task correctly, used the right tool, respected policy, and stayed within delegated authority. That is why failure detection needs structured evals and replayable traces, not just free-text search.

Which failure modes keyword search tends to miss

Keyword search tends to miss failures that do not produce distinctive language. Tool misuse can appear as a legitimate action sequence until you compare it with the intended workflow. Stale context can cause an agent to act on outdated facts without any obvious keyword anomaly. Overstepped delegation can look like ordinary productivity until the trace shows the agent crossing an approval boundary.

It also misses composite failures, where several small issues add up to a bad result. An agent may retrieve plausible context, choose an acceptable-looking tool, and still violate policy because the state was stale or the task was routed through the wrong branch. Searching for a single term rarely catches those multi-step failures, because the defect is behavioural and distributed across the run.

That is why the most useful failure signals are usually outcome-based: did the agent finish the task, did it make the right sequence of decisions, did it call the correct tools, and did it avoid actions that should have been blocked. The strongest evaluations compare expected behaviour to observed traces, then score the gap directly.

How to evaluate agent failures from traces instead of terms

Use evals that can score the agent at several levels at once. Task completion tells you whether the overall objective was met. Tool-use scoring tells you whether the agent chose and sequenced actions correctly. Policy-compliance scoring shows whether the run respected guardrails, approvals, and prohibited actions. Failure-mode scoring helps you label the specific class of defect so you can fix the right layer.

This is especially important when the same agent can fail in more than one way. A run may complete the task but still expose a control weakness, or it may fail without an obvious textual clue. A trace-based eval can capture both because it inspects action history, not just the final output.

For this reason, teams should prefer replayable traces and structured criteria over ad hoc string matching. The more the evaluation depends on behaviour, state transitions, and tool results, the less it depends on the agent using a predictable vocabulary.

Risk and Threat Considerations

Keyword-based monitoring creates blind spots in systems where the material risk is behavioural drift rather than obvious malicious language. A failure can still create real exposure if the agent makes an incorrect external action, discloses data through the wrong tool, or proceeds after authority should have stopped it. That makes the monitoring gap itself a security problem.

Failure mechanism: The agent can satisfy a text query while still choosing the wrong tool, acting on stale context, or exceeding delegated authority, so the risky behaviour never appears as a searchable keyword pattern.

Impact: Defects can persist in production, because the control only sees language similarity rather than task execution, policy compliance, and side effects. That increases the chance of undetected misuse, bad automation, and delayed response to harmful agent behaviour.

Practitioner Guidance

What to prioritise: Start with the agent actions that can cause external impact, not with the text that the model emits. Review tool calls, state changes, approvals, and exceptions before you tune any keyword rules.

What to verify: A useful eval should tell you not only whether the answer was correct, but whether the agent followed the intended path to get there. If the trace does not let you inspect tool choice, policy checks, and final outcome together, the eval is too shallow for failure detection.

Decision rule: If a failure would matter even when the output text looks harmless, treat keyword search as a triage aid only. Use trace-based scoring for the real control, because that is what reveals silent behavioural failures.

Practitioner takeaway: For agents, the important question is usually “what did it do?” rather than “what words did it use?”, so your evaluation strategy should follow the behaviour, not the vocabulary.

Related resources from NHI Mgmt Group

• Why do synthetic test sets miss the most important agent failures?
• Why is NHI ownership attribution important for incident response?
• Why is visibility important in managing Shadow AI?
• Why is visibility important for Non-Human Identities?