Notifications
Clear all
Topic starter
21/08/2026 4:53 pm
TL;DR: RAG monitoring still fails when teams focus on model output alone, because the real risk sits in prompts, retrieved context, PII leakage, and unsafe responses flowing through production traces, according to Fiddler. Identity and access controls matter here because the application can expose sensitive data and credentials through the retrieval layer, not just the model layer.
NHIMG editorial — based on content published by Fiddler: How to Monitor Your DataStax RAG Applications with Fiddler
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing access credentials.
Questions worth separating out
Q: How should security teams monitor RAG applications in production?
A: Monitor the full trace, not only the final answer.
Q: Why do RAG systems create a bigger privacy risk than chat models alone?
A: RAG systems can surface sensitive information from retrieved documents even when the user never directly asked for it.
Q: What breaks when prompt and retrieval telemetry are missing?
A: Teams lose the ability to prove what the model saw, why it answered, or whether sensitive content was introduced by retrieval.
Practitioner guidance
- Instrument full RAG trace capture Capture prompts, retrieved documents, responses, metadata, and timestamps for every production interaction so investigations can reconstruct the full decision path.
- Classify sensitive retrieval paths Tag data sources and retrieval scopes by sensitivity so teams can spot when chatbots are able to surface secrets, personal data, or regulated content.
- Wire observability into incident workflows Route prompt-safety, PII leakage, and jailbreak alerts into existing privacy and security response processes so AI events are handled like operational incidents.
What's in the full article
Fiddler's full technical integration tutorial covers the implementation detail this post intentionally leaves for the source:
- Next.js API route code for publishing LLM events into the observability platform
- Fiddler client setup and model onboarding steps for defining the application schema
- Trust score configuration for prompt safety, toxicity, answer relevance, and PII leakage
- Dashboard and diagnostic workflow examples for tracing hallucinations and unsafe responses
👉 Read Fiddler's technical integration tutorial for monitoring DataStax RAG applications →
RAG monitoring for AI observability teams: what changes in practice?
Explore further
21/08/2026 5:11 pm
RAG observability is becoming a governance control, not just a model-quality tool. The article shows that prompt, retrieval, and response telemetry are now part of the control surface for AI systems. That matters because security teams need evidence of what the model saw, not just what it answered. Practitioners should treat observability data as governed security telemetry, especially where prompts or retrieved context can contain secrets or personal data.
A question worth separating out:
Q: Who should own RAG safety incidents: AI teams, security teams, or privacy teams?
A: Ownership should be shared, but security should run containment, AI teams should explain model behavior, and privacy teams should assess disclosure impact. The key is to define escalation before production use so prompt injection, PII leakage, and unsafe outputs are handled through a clear accountability chain.
👉 Read our full editorial: Monitoring RAG applications needs identity-aware AI observability
