Dual-agent search benchmarks: what do current models still miss?

TL;DR: Current state-of-the-art models struggle with multi-turn, underspecified retrieval, reaching only 36% accuracy across text, image, audio and video, while the best model stays below 70% and every model performs worse once image inputs are added, according to Patronus AI’s DETOUR benchmark. The result shows how brittle agentic search remains when memory, context and modality shifts are part of the task, not the exception.

Editorial analysis by NHI Mgmt Group, based on content published by Patronus AI: “DETOUR”.

By the numbers:

• The full dual-agent harness, text-only and with image inputs, reports mean results from three runs with 95% confidence intervals.
• Current state-of-the-art models achieve only 36% accuracy when evaluated on all modalities.

Key questions

Q: How should teams evaluate agentic systems that rely on multi-turn retrieval?

A: Evaluate them on underspecified, multi-turn tasks that require the system to infer an entity from partial clues, then test the same workflow with and without extra modalities.

Q: Why do multimodal inputs make agent search and reasoning less reliable?

A: Additional modalities can add noise, distractors and alignment problems instead of clarity.

Q: What breaks when an agent cannot identify the right entity from partial context?

A: The failure is usually not just a wrong answer.

Practitioner guidance

• Add underspecified retrieval tests to agent evaluation Measure multi-turn search tasks where the system must infer the target from partial clues, then compare text-only and multimodal performance under the same rubric.
• Gate tool use on retrieval confidence Require the agent to prove it has enough evidence before invoking tools, escalating privileges or continuing a workflow that depends on inferred identity.
• Instrument modality-specific failure signals Log where image, audio or video inputs reduce certainty so you can see whether the model is failing at grounding, fusion or entity resolution.

Bottom line: DETOUR shows that agentic search remains brittle when the right answer must be inferred across turns and modalities, not selected from a clean prompt.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →