Interactive dual-agent search still fails under underspecified prompts

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Patronus AI's research: DETOUR shows that dual-agent, multi-turn retrieval remains unreliable, with current models reaching only 36% accuracy across all modalities.

Why it matters: For teams building AI agents, this matters because weak retrieval under underspecified prompts can cascade into bad tool use, poor delegation decisions, and ungoverned downstream actions.

By the numbers:

• The full dual-agent harness, text-only and with image inputs, reports mean results from three runs with 95% confidence intervals.
• The best model stays below 70% accuracy in the benchmark.
• Current state-of-the-art models achieve only 36% accuracy when evaluated on all modalities.

Context

DETOUR is a benchmark for testing how an AI system behaves when it must infer the right answer from incomplete, conversational clues rather than a clean one-shot prompt. That matters because many real agent workflows depend on partial recall, iterative querying and cross-modal evidence, not static question answering.

The identity angle is less about human authentication and more about how agentic systems earn the right to act on uncertain information. When retrieval is weak, delegation, tool choice and memory use become governance problems, not just model-quality issues.

Key questions

Q: How should teams evaluate agentic systems that rely on multi-turn retrieval?

A: Evaluate them on underspecified, multi-turn tasks that require the system to infer an entity from partial clues, then test the same workflow with and without extra modalities. The key question is not only whether the model is correct, but whether it remains reliable enough to justify tool use, delegation and policy enforcement when the answer is incomplete.

Q: Why do multimodal inputs make agent search and reasoning less reliable?

A: Additional modalities can add noise, distractors and alignment problems instead of clarity. If the model cannot ground image, audio or video evidence consistently, its retrieval confidence drops and so does the quality of any downstream decision based on that retrieval. Multimodal systems need evidence scoring, provenance and fallback paths before they act.

Q: What breaks when an agent cannot identify the right entity from partial context?

A: The failure is usually not just a wrong answer. The agent may attach the wrong identity, choose the wrong tool, or continue a workflow under an assumption that was never validated. That turns a retrieval error into a governance error because the system may act with inappropriate confidence.

Q: When should organisations require human approval for an AI agent action?

A: Require human approval when the action could change infrastructure, expose sensitive data, move laterally across systems, or trigger a business-critical workflow that is hard to reverse. Approval is also warranted when the agent’s decision depends on ambiguous input or external data that cannot be trusted at face value. High-consequence actions need a human stop point.

Technical breakdown

Why dual-agent retrieval is harder than single-turn evaluation

DETOUR uses a Primary Agent that must identify a recollected entity by querying a Memory Agent held constant across evaluations. That design is closer to real conversational search than static benchmarks because the system must use dialogue state, not just pattern match a prompt. The challenge is underspecification: the model has to resolve ambiguity through iterative questioning, cross-turn context and modality fusion. When the same task is repeated across text, image, audio and video, the search space expands and confidence calibration becomes part of the problem, not a side effect. See the Agentic AI Identity Guide.

Practical implication: test agent workflows under multi-turn ambiguity, not only single-turn correctness.

Why image inputs change the failure mode

The benchmark shows that adding image inputs reduces accuracy for every model, which suggests the weakness is not merely textual recall. Visual context can introduce distractors, weak grounding or inconsistent cross-modal alignment, especially when the agent must infer an entity from partial clues. In practice, this is an integration problem as much as a modelling problem: the system needs reliable modality fusion, clear provenance for retrieved evidence and disciplined handling of uncertainty before it acts on an inferred match. See the AI Agent Observability, Audit and Incident Response Guide.

Practical implication: treat multimodal retrieval as a separate control surface, not an extension of text-only prompting.

How agent identity and delegated authority depend on retrieval quality

When an agent cannot reliably recover the right entity from underspecified context, identity and authorisation decisions become fragile. That matters for agentic systems because the wrong retrieval can map to the wrong user, workload, document or tool, and the system may then issue actions under an incorrect assumption. In governance terms, the problem sits at the boundary between memory, policy and delegated authority. A model that cannot ground its recall should not be trusted to escalate privileges, invoke tools or continue a workflow without additional checks. See the AI Agent Authorisation Guide.

Practical implication: bind retrieval confidence to policy enforcement and step-up approval.

NHI Mgmt Group analysis

DETOUR exposes a retrieval trust gap, not just a model accuracy gap: the benchmark shows that current systems still struggle when the answer is spread across turns and modalities. That means the control problem is not only generation quality but whether the agent can ground its next action in evidence strong enough to justify delegation. For practitioners, the implication is straightforward: if retrieval is uncertain, downstream autonomy is premature.

Multimodal agent workflows need confidence gating before action: the drop in performance with image inputs shows that adding modalities can increase ambiguity as easily as it increases context. In agentic programmes, this is where governance should move from prompt design to evidence quality, provenance and threshold-based escalation. observability and incident response for agents becomes relevant when you need to understand why a system acted on the wrong clue.

Agent identity becomes unreliable when memory is weak: if an agent cannot consistently recover the intended entity, then the system may attach the wrong identity, permission set or tool context to the task. That is a governance failure because the authorisation decision is only as good as the retrieval that precedes it. Practitioners should treat identity binding, not just authentication, as part of the benchmark surface.

Named concept, retrieval ambiguity debt: this benchmark illustrates the accumulated risk that emerges when systems are evaluated on neat prompts but deployed into messy, under-specified interactions. The debt shows up as bad recall, poor modality fusion and overconfident action. Teams should measure whether their agent stack can resolve ambiguity before they let it operate across tools or workflows.

Read next: Agentic AI Identity Maturity Model

What this signals

Retrieval ambiguity debt: agentic systems accumulate risk when they are trained and benchmarked on clean prompts but deployed into messy, underspecified interactions. The operational lesson is to move confidence checks upstream, before a retrieval miss becomes a delegated action.

Practitioner programmes should treat multimodal search as a control boundary. If image inputs consistently reduce certainty, the issue is not just model performance. It is evidence handling, and that makes agent observability and incident response relevant to governance and response design.

For practitioners

• Add underspecified retrieval tests to agent evaluation Measure multi-turn search tasks where the system must infer the target from partial clues, then compare text-only and multimodal performance under the same rubric.
• Gate tool use on retrieval confidence Require the agent to prove it has enough evidence before invoking tools, escalating privileges or continuing a workflow that depends on inferred identity. See the AI Agent Authorisation Guide.
• Instrument modality-specific failure signals Log where image, audio or video inputs reduce certainty so you can see whether the model is failing at grounding, fusion or entity resolution.
• Separate evaluation of memory and action Assess whether the agent can recover the right entity before you judge whether it should be allowed to decide, delegate or execute.

Key takeaways

• DETOUR shows that agentic search remains brittle when the right answer must be inferred across turns and modalities, not selected from a clean prompt.
• The reported 36% accuracy across all modalities and sub-70% ceiling for the best model indicate a persistent governance gap, not a corner-case failure.
• Teams should gate tool use and delegated action on retrieval confidence, because uncertain recall can turn into incorrect identity binding and unsafe execution.

Related NHI Mgmt Group resources: Agentic AI Identity Guide · Agentic AI Identity Maturity Model · AI Agent Authorisation Guide · AI Agent Observability, Audit and Incident Response Guide

Key terms

• Dual-Agent Evaluation: A testing pattern in which one agent is evaluated while another agent supplies or mediates information during the task. It is useful for studying conversational search, delegation and uncertainty because the evaluated system must reason over interactive evidence rather than a fixed prompt alone.
• Underspecified Retrieval: A retrieval task where the target cannot be identified from a single clear clue and must be inferred from partial, ambiguous or multi-turn context. In agentic systems, this stresses grounding, evidence quality and confidence calibration before the system takes action.
• Modality Fusion: The process of combining text, image, audio and video evidence into a single decision path. When fusion is weak, a system may overfit one input type, lose confidence or draw the wrong conclusion from mixed signals, especially in interactive search workflows.
• Retrieval Confidence: The system’s degree of certainty that it has identified the correct entity, answer or evidence set. In governance terms, confidence should determine whether the agent can continue autonomously, request more context or hand off to a human reviewer.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need to connect agent identity, access decisions, and governance controls across real programmes.