Notifications
Clear all
Topic starter
20/08/2026 5:54 pm
TL;DR: An 8-hour autonomous loop in which Claude Code, guided by Arize evaluation tooling, raised Recall@5 from 39% to 75% by iterating on indexing, retrieval, and reranking changes across 17 experiments, according to Arize. The result shows that retrieval quality, not prompt tuning, remains the ceiling for RAG performance, and that measurable feedback loops are now central to AI system governance.
NHIMG editorial — based on content published by Arize: How Arize Skills Improved RAG Recall from 39% to 75% in 8 Hours
By the numbers:
- Arize reports that Recall@5 climbed from 39% to 75% in 8 hours during the self-improvement loop.
- Arize says the loop ran through 17 experiments with consistent metrics, allowing direct comparison across iterations.
Questions worth separating out
Q: How should security teams govern autonomous RAG improvement loops?
A: Security teams should require explicit change boundaries, benchmark gating, and rollback paths before allowing autonomous loops to modify retrieval, ranking, or indexing logic.
Q: Why do retrieval metrics matter for AI risk management?
A: Retrieval metrics matter because they define the ceiling for answer quality.
Q: What breaks when autonomous agents can change retrieval logic on their own?
A: What breaks is change accountability.
Practitioner guidance
- Implement benchmark-gated promotion for retrieval changes Require every change to chunking, reranking, embeddings, or query expansion to pass the same evaluation set before it can reach production.
- Separate live and experimental indexes Use versioned indexes with atomic alias swaps so retrieval changes can be promoted or rolled back without rewriting the active environment.
- Record change provenance for every autonomous iteration Log which agent action changed the code, which metric moved, and which dataset was used so performance deltas remain explainable.
What's in the full article
Arize's full article covers the operational detail this post intentionally leaves for the source:
- The exact iteration flow used inside Claude Code, including how evaluation results fed the next change set.
- The code-level split between index and agent improvements, which shows how the team isolated retrieval changes from model-side changes.
- The experiment progression behind each recall jump, including which tuning choices helped and which ones were abandoned.
- The reproducible setup details for Arize Skills, OpenSearch, and the self-rag project structure.
👉 Read Arize's analysis of autonomous RAG evaluation and recall improvement →
Autonomous RAG evaluation loops: what do practitioners need to know?
Explore further
20/08/2026 6:34 pm
Retrieval governance is now part of AI security governance. Once an AI system can change retrieval, reranking, or chunking based on its own evaluation loop, the security question shifts from prompt safety to change control. That is especially relevant in environments where AI systems touch sensitive documents, access decisions, or workflow routing. In NHIMG terms, this is where AI governance starts to overlap with IAM discipline, because the system is effectively deciding what information is visible to whom or what. Practitioners should treat retrieval pipelines as governed security components, not just model plumbing.
A question worth separating out:
Q: How can organisations tell whether AI self-improvement is under control?
A: Look for repeatable metrics, immutable experiment logs, versioned artifacts, and atomic promotion between experimental and live environments. If the system cannot show what changed, why it changed, and how to reverse it, the self-improvement loop is not controlled enough for production use.
👉 Read our full editorial: Autonomous RAG evaluation shifts recall from 39% to 75% in 8 hours
