Notifications
Clear all
Topic starter
18/08/2026 2:49 pm
TL;DR: Reliable RAG systems need continuous evaluation, because retrieval failures and generation failures can be isolated, measured, and improved separately, according to HoneyHive’s guide with MongoDB. The operational lesson is that AI production quality depends on disciplined testing, traceability, and iteration rather than one-time prompt tuning.
NHIMG editorial — based on content published by HoneyHive: Towards Evaluation Driven Development with MongoDB and HoneyHive
By the numbers:
- 71% of NHIs are not rotated within recommended time frames, increasing the risk of compromise over time.
Questions worth separating out
Q: How should security teams govern access in RAG systems?
A: Security teams should govern RAG access at the retrieval layer, not only at authentication.
Q: Why do RAG systems need continuous evaluation instead of one-time testing?
A: RAG systems change as data, prompts, embeddings, and retrieval settings change, so a passing test today can become a failure tomorrow.
Q: What breaks when retrieval quality is not measured separately from model output quality?
A: When retrieval and generation are blended together, teams cannot tell whether bad answers came from the wrong context or from poor reasoning over good context.
Practitioner guidance
- Separate retrieval and generation metrics Track context relevance, answer relevance, and answer faithfulness as distinct indicators so you can identify whether the failure sits in retrieval, prompting, or model behaviour.
- Treat chunking and top-k as governed controls Document chunk size, chunk overlap, embedding model choice, and similarity_top_k values as part of the production baseline.
- Build a regression dataset from real failure cases Capture failed traces, convert them into repeatable evaluation questions, and rerun them after every prompt, index, or model change.
What's in the full article
HoneyHive's full guide covers the operational detail this post intentionally leaves for the source:
- End-to-end setup for the MongoDB Atlas Vector Search index and LlamaIndex integration.
- Concrete HoneyHive evaluator configuration for context relevance, answer faithfulness, and answer relevance.
- Tracing and run-management workflow examples that show how to capture evaluation sessions in practice.
- Dataset-generation code for producing test questions from source documents and replaying them in the harness.
👉 Read HoneyHive's guide to evaluation driven development for production RAG →
RAG evaluation driven development: what practitioners need to change?
Explore further
18/08/2026 3:31 pm
Evaluation discipline is becoming an identity governance issue for AI systems. RAG pipelines often move sensitive content through retrieval layers, prompt inputs, and observability tools that behave like governed access paths even when teams do not treat them that way. Once an AI workflow can surface internal knowledge, secrets handling, trace retention, and data minimisation become part of the control surface. Practitioners should govern AI pipelines as access-bearing systems, not just application features.
A question worth separating out:
Q: Which identity and governance controls matter when AI systems access sensitive knowledge bases?
A: The same controls that protect production systems matter here: least privilege for data sources, trace retention limits, secret handling, change approval, and clear ownership for evaluation results. If an AI pipeline can expose internal data, it must be governed as a system with access boundaries, not treated as a neutral interface.
👉 Read our full editorial: Evaluation driven development for RAG systems needs governance
