Topic starter
20/08/2026 5:54 pm
TL;DR: An 8-hour autonomous loop in which Claude Code, guided by Arize evaluation tooling, raised Recall@5 from 39% to 75% by iterating on indexing, retrieval, and reranking changes across 17 experiments, according to Arize. The result shows that retrieval quality, not prompt tuning, remains the ceiling for RAG performance, and that measurable feedback loops are now central to AI system governance.
Editorial analysis by NHI Mgmt Group, based on content published by Arize: “How Arize Skills Improved RAG Recall from 39% to 75% in 8 Hours”.
Key questions
Q: What breaks when retrieval recall stays below acceptable thresholds in RAG systems?
A: When retrieval recall stays low, the generator is forced to work with incomplete evidence and cannot reliably answer even if the model itself is strong.
Q: Why do autonomous evaluation loops matter for AI system governance?
A: They matter because the system can continuously alter its own code, retrieval configuration, and experiment backlog based on results.
Q: How should teams decide whether to tune prompts or improve retrieval first?
A: Improve retrieval first when the failure mode is missing context, weak chunking, poor ranking, or near-miss recall.
Practitioner guidance
- Establish Recall@k as a release gate Define Recall@1, Recall@5, and Recall@10 thresholds for retrieval changes before any model or index promotion.
- Separate retrieval tuning from prompt tuning Review chunking, indexing, embedding selection, and reranking before iterating on prompts.
- Make experiment provenance mandatory Record every evaluation run, dataset version, and ranking change so the team can trace which modification produced a score shift.
Bottom line: The article shows that RAG performance is constrained by what retrieval returns, not by how persuasive the generator sounds.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:34 pm
Closed-loop evaluation is becoming the governance boundary for AI systems: once an agent can measure itself, propose changes, and rerun experiments, oversight moves from review of static code to governance of the improvement loop. That shifts the control question from "is the model accurate?" to "is the system allowed to rewrite the conditions under which accuracy is measured?" Practitioners should treat the loop itself as a governed asset.
A question worth separating out:
Q: How should organisations govern agent access to indexing and evaluation tools?
A: Treat those tools as privileged capabilities and restrict them to approved workflows with logging, approval, and rollback. An agent that can rerun experiments, rewrite retrieval logic, or promote index versions can materially change production behaviour, so access control should be as strict as for other sensitive production actions.
👉 Read our full editorial: Autonomous RAG evaluation shifts recall from 39% to 75% in 8 hours