Production RAG needs observability, evaluation and routing controls

At a glance

What this is: This is an analysis of how Google ADK and Arize AX are used to operationalise production RAG with orchestration, observability, and evaluation.

Why it matters: It matters because RAG systems now behave like governed AI services, so IAM, NHI, and AI security teams need visibility into tool use, data access, and answer quality.

By the numbers:

• Only 5.7% of organisations have full visibility into their service accounts.
• 96% of organisations store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools.
• 90% of IT leaders say properly managing NHIs is essential for a successful zero-trust implementation.

👉 Read Arize's analysis of production RAG orchestration, observability and evaluation

Context

Production RAG is less a prompt-engineering problem than a governance problem. Once retrieval, routing, citations, and response generation are chained together, failures can come from the model, the knowledge corpus, the tool layer, or the way identity and access are granted to each component. That makes the topic relevant to both AI operations and identity governance.

The identity intersection is real wherever RAG agents query documents, APIs, and vector stores on behalf of users or workloads. In those flows, service accounts, API keys, and access scopes become part of the system’s trust boundary, and poor lifecycle control can turn a quality problem into a data exposure problem.

For most enterprises, this is a typical maturity gap: the architecture exists before the control model does. The result is a working pilot that cannot yet explain, prove, or constrain its own answers in production.

Key questions

Q: How should security teams govern production RAG systems with private data sources?

A: Treat production RAG as a governed access path, not just an application feature. Separate the identities used for retrieval, tracing, and generation, restrict each to the minimum data scope, and log every query to private sources. The goal is to prevent a model-quality issue from becoming a broader data access failure.

Q: Why do RAG systems still hallucinate when retrieval is enabled?

A: Retrieval does not eliminate hallucination when the retrieved context is irrelevant, stale, or only loosely connected to the question. The model can still synthesise unsupported claims from weak evidence. Teams need citation verification, groundedness checks, and corpus quality controls to reduce that failure mode.

Q: What breaks when RAG observability is missing?

A: Without RAG observability, teams can see that a service is healthy while missing the real failure in retrieval, reranking, or context assembly. That leads to confident but wrong answers, slower debugging, and a false sense of control because the model output looks plausible even when the evidence chain is broken.

Q: How do teams know if RAG quality is actually improving?

A: Teams know quality is improving when live scores for groundedness, faithfulness, and answer relevance trend upward across real traffic, not just benchmark sets. They should also check whether long-tail cohorts improve, because aggregate gains can hide regressions in specific query classes.

Technical breakdown

Production RAG orchestration and retrieval routing

Production RAG is not a single search step followed by generation. It is a control flow that decides which source to query, how many chunks to retrieve, how to rank them, and when to trust the retrieved context enough to answer. Agentic orchestration matters because different queries need different retrieval paths, and without routing discipline, the system overuses irrelevant context or misses authoritative sources. This becomes a governance issue when the same application can touch documents, APIs, and private data stores under one runtime identity.

Practical implication: define retrieval scopes, source priorities, and identity boundaries before the agent is allowed to query production data.

Why citation verification and groundedness monitoring matter

RAG reliability depends on whether answers are grounded in retrieved evidence, not merely fluent. Citation verification checks whether the cited material actually supports the output, while groundedness measures whether the response stayed within the retrieved context. These controls matter because a system can appear correct while quietly hallucinating details or stitching together weakly relevant fragments. In enterprise settings, that is a data governance and assurance issue as much as a model quality issue.

Practical implication: monitor groundedness and citation fidelity as production controls, not post-incident diagnostics.

Context quality, embeddings, and hidden knowledge gaps

The quality of retrieval is limited by the quality of the corpus, chunking strategy, embeddings, and metadata attached to documents. If the knowledge base is stale, incomplete, or poorly segmented, the model can only produce confident answers to the wrong evidence. Embedding analysis helps expose topic clusters, drift, and missing documentation, which is especially useful when RAG is used for internal knowledge systems that change faster than the retrieval layer is updated.

Practical implication: treat corpus curation and metadata hygiene as part of the control plane, not a content team side task.

NHI Mgmt Group analysis

Production RAG creates an identity and access problem hidden inside an AI quality problem. The article shows that RAG reliability depends on more than retrieval accuracy, because the system must decide which knowledge sources can be queried, by whom, and under what scope. That is where IAM, service-account governance, and workload identity become part of AI assurance. Practitioners should treat RAG access paths as governed identities, not invisible implementation detail.

Observability is now the missing control layer for enterprise RAG. Traditional monitoring tells teams whether a service is up, but not whether retrieval was relevant, whether the context was used correctly, or whether the answer was supported by evidence. That gap creates a blind spot between model behaviour and operational accountability. Practitioners should demand traceability from query to citation, because without it, production RAG cannot be audited or defended.

Context quality is the new version of data quality in AI systems. The article’s focus on retrieval quality, embedding analysis, and corpus preparation shows that RAG failures often originate before generation begins. Retrieval trust gap: when the corpus, access model, and citation chain are not aligned, the system can appear authoritative while drawing from weak or stale evidence. Practitioners should align data governance, document lifecycle, and access control with the retrieval layer.

AI governance for RAG must include workload identity and secrets discipline. Any agent that queries private repositories, vector stores, or cloud APIs is operating through credentials, tokens, or service accounts. That means secrets leakage, over-broad access, and poor offboarding can expose the RAG stack itself, not just its outputs. Practitioners should govern RAG agents as privileged workloads with explicit lifecycle controls.

The market is moving from pilot success metrics to production assurance metrics. The article reflects a broader shift in enterprise AI: the question is no longer whether a demo works, but whether the system can be evaluated, traced, and improved continuously. That shift aligns with NIST AI RMF thinking on measurement and governance. Practitioners should benchmark systems on groundedness, citation accuracy, and retrieval coverage rather than anecdotal user satisfaction.

What this signals

Production RAG will increasingly be judged on traceability and governance, not just answer quality. That means teams should expect more scrutiny of who can query which knowledge sources, how citations are verified, and how retrieval failures are detected before they reach users.

Retrieval trust gap: organisations that cannot prove the link between retrieved context and generated output will struggle to operationalise RAG beyond low-risk use cases. The practical response is to connect evaluation, document lifecycle, and access control into one assurance model.

The work also pushes identity teams closer to AI operations, because any agent that reaches into internal repositories does so through service accounts and secrets. That is where RAG reliability and identity governance become the same control conversation.

For practitioners

• Separate orchestration from authorization Assign distinct identities and access scopes to retrieval services, embedding pipelines, and generation workloads so a failure in one layer cannot expose every data source.
• Instrument citation and groundedness checks Track whether each answer can be traced to retrieved evidence, and alert when citations no longer support the response or when the model departs from the source context.
• Curate the corpus as a governed asset Review chunking, metadata, document freshness, and source prioritisation together so retrieval quality does not depend on ad hoc content hygiene.
• Control secrets used by RAG workloads Inventory API keys, service accounts, and tokens attached to retrieval and tracing components, then rotate or revoke them on a defined lifecycle rather than by incident.
• Use evaluation results to drive remediation Turn hallucination, relevance, and answer-correctness failures into a repeatable improvement loop that updates prompts, retrieval settings, and corpus coverage.

Key takeaways

• Production RAG fails most often when orchestration, retrieval quality, and answer verification are treated as separate problems.
• The operational gap is traceability: teams need to prove what context was used, how it was selected, and whether the answer stayed grounded.
• Identity, secrets, and workload access now sit inside the RAG control plane, so governance has to extend beyond the model layer.

Key terms

• Retrieval-augmented Generation: Retrieval-augmented generation is a pattern where an AI model pulls external information before generating output. The security challenge is that access rules can weaken when data is chunked, embedded, cached, or reused, so source permissions may not automatically follow the content into the model's context.
• Groundedness: Groundedness is the degree to which an AI response can be supported by verifiable source material. In practice, it measures whether the model answered from evidence rather than inference, memory, or fabrication, which is critical for RAG systems and any workflow that drives decisions from model output.
• Citation Verification: A validation check that confirms whether the cited source actually supports the generated answer. It helps distinguish answers that are merely well formatted from answers that are genuinely evidence based, which is essential when RAG systems are used for internal decision support.
• Retrieval Trust Boundary: The point in an AI workflow where externally sourced content becomes part of the model's decision context. When that boundary is weak, unvalidated data can shape outputs, leak sensitive information, or override intended guardrails, making governance fail at the input stage.

What's in the full article

Arize's full blog covers the operational detail this post intentionally leaves at the governance level:

• Step-by-step tracing setup for ADK instrumention and production observability inside Arize AX
• Concrete evaluator examples for hallucination, groundedness, relevancy, and citation verification
• Implementation details for automatic online evaluation and regression dataset curation
• Embeddings analysis workflows that reveal knowledge-base drift and retrieval gaps

👉 The full Arize post covers tracing setup, evaluator examples, and embedding-based debugging for RAG systems.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It gives identity and security practitioners a practical way to apply lifecycle controls to AI-enabled systems.