Notifications
Clear all
Topic starter
18/08/2026 2:16 pm
TL;DR: RAG pipelines break at different stages, so end-to-end scores alone can hide whether retrieval, context assembly, or generation caused a regression, according to Braintrust. The practical lesson is that teams need representative datasets, stage-level scoring, and release gates before shipping changes.
NHIMG editorial — based on content published by Braintrust: How to test your RAG pipeline (before and after you ship)
By the numbers:
- Braintrust recommends starting with 5 to 10 representative examples that cover core use cases.
Questions worth separating out
Q: How should security teams test a RAG pipeline before release?
A: Test the retriever, context assembly, and generator separately before you trust the full pipeline.
Q: Why do RAG systems need separate retrieval and generation scoring?
A: Because a fluent answer can still be built on weak evidence, and a retrieval regression can be masked by a stronger prompt or model.
Q: How can security and AI teams know if RAG evaluation is actually working?
A: Evaluation is working when test results change predictably after controlled updates and when failures map to a specific component, such as retrieval, chunking, or the generation layer.
Practitioner guidance
- Instrument retrieval and generation separately Record retrieval and generation as distinct spans in the same trace so you can score document selection and answer quality independently.
- Build a holdout dataset from production queries Use production logs to capture real phrasing, ambiguity, and edge cases, then reserve a holdout set for release checks that should not be used during tuning.
- Pin every evaluation dependency Store corpus or index version, document-processing settings, embedding model, retrieval parameters, reranker configuration, generation model, and prompt version with each run.
What's in the full article
Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:
- Step-by-step examples for building golden datasets from production logs and holdout cases.
- Metric selection guidance for recall@k, precision@k, NDCG, ContextRecall, and AnswerCorrectness.
- CI workflow patterns for smoke testing pull requests and running full release checks after merge.
- Trace and scoring examples that show how to pin corpus, prompt, and model versions across experiments.
👉 Read Braintrust's guide to testing a RAG pipeline before and after shipping →
RAG testing: are retrieval and generation being measured separately?
Explore further
18/08/2026 3:07 pm
Separate evaluation is now a governance requirement, not a testing preference. The article shows that end-to-end RAG scoring can conceal whether the retriever, the context assembly layer, or the generator caused the failure. That is a control problem, not just a measurement problem, because teams cannot govern what they cannot isolate. For identity and access teams, the parallel is clear: the system that fetches evidence needs the same scrutiny as the system that answers with it. The practitioner conclusion is to treat stage-level evaluation as part of model governance.
A question worth separating out:
Q: What should teams do when a RAG change improves the average score but breaks key cases?
A: Prioritise the critical cases. Average scores can hide concentrated regressions in authentication, policy, or high-risk operational queries. A release should be blocked if must-pass cases fail, even if the aggregate score moves up. Good governance means deciding in advance which failures are unacceptable and enforcing that rule consistently.
👉 Read our full editorial: RAG pipeline testing fails when retrieval and generation blur
