TL;DR: Dual-agent search still struggles across multi-turn, underspecified retrieval, reaching only 36% accuracy and staying below 70% at best according to Patronus AI. Retrieval systems need tighter confidence checks before tool use or delegated action can turn a weak match into a wrong decision.
Editorial analysis by NHI Mgmt Group, based on content published by Patronus AI: “DETOUR”.
By the numbers:
- The full dual-agent harness, text-only and with image inputs, reports mean results from three runs with 95% confidence intervals.
- Current state-of-the-art models achieve only 36% accuracy when evaluated on all modalities.
Key questions
Q: Why do multimodal inputs make agent search and reasoning less reliable?
A: Additional modalities can add noise, distractors and alignment problems instead of clarity.
Q: What breaks when an agent cannot identify the right entity from partial context?
A: The failure is usually not just a wrong answer.
Q: How should teams evaluate agentic systems that rely on multi-turn retrieval?
A: Evaluate them on underspecified, multi-turn tasks that require the system to infer an entity from partial clues, then test the same workflow with and without extra modalities.
Practitioner guidance
- Add underspecified retrieval tests to agent evaluation Measure multi-turn search tasks where the system must infer the target from partial clues, then compare text-only and multimodal performance under the same rubric.
- Gate tool use on retrieval confidence Require the agent to prove it has enough evidence before invoking tools, escalating privileges or continuing a workflow that depends on inferred identity.
- Instrument modality-specific failure signals Log where image, audio or video inputs reduce certainty so you can see whether the model is failing at grounding, fusion or entity resolution.
Bottom line: DETOUR shows that agentic search remains brittle when the right answer must be inferred across turns and modalities, not selected from a clean prompt.
Glossary terms: Dual-Agent Evaluation, Underspecified Retrieval, Modality Fusion, Retrieval Confidence
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
DETOUR exposes a retrieval trust gap, not just a model accuracy gap: the benchmark shows that current systems still struggle when the answer is spread across turns and modalities. That means the control problem is not only generation quality but whether the agent can ground its next action in evidence strong enough to justify delegation. For practitioners, the implication is straightforward: if retrieval is uncertain, downstream autonomy is premature.
A question worth separating out:
Q: When should organisations require human approval for an AI agent action?
A: Require human approval when the action could change infrastructure, expose sensitive data, move laterally across systems, or trigger a business-critical workflow that is hard to reverse. Approval is also warranted when the agent’s decision depends on ambiguous input or external data that cannot be trusted at face value. High-consequence actions need a human stop point.
👉 Read our full editorial: Interactive dual-agent search still fails under underspecified prompts