How do you know whether an LLM failure came from retrieval or generation?

Why This Matters for Security Teams

When an LLM answer is wrong, teams need to know whether the issue began in retrieval, prompt assembly, model choice, or generation. That distinction changes the fix, the owner, and the evidence to preserve. A clean retrieval trace with a flawed answer points to model behaviour, while stale or incomplete context points to data quality, indexing, or routing. The NIST AI 600-1 Generative AI Profile is useful here because it treats traceability, measurement, and output quality as governance concerns, not just engineering details.

Security teams often get this wrong by looking only at the final response text. That misses the operational evidence needed to isolate failure domains in a retrieval-augmented workflow. In practice, the most expensive mistakes are usually not model hallucinations alone, but unchecked assumptions that the right context was actually available to the model at inference time. This becomes more serious when the system has tool access or agentic routing, because a bad answer may also reflect unsafe orchestration, not just language generation.

Current guidance suggests treating retrieval and generation as separate control points in the incident path. That means preserving span order, prompt payloads, retrieved chunks, ranking scores, and model metadata so the failure can be triaged with confidence. In practice, many security teams encounter the root cause only after users have already acted on a wrong answer, rather than through intentional trace review.

How It Works in Practice

The quickest way to separate retrieval failure from generation failure is to inspect the execution trace from left to right. First confirm that the query was interpreted correctly, then check whether the retriever returned relevant passages, and finally verify that those passages were actually injected into the prompt sent to the model. If the retrieval span is weak, empty, stale, or semantically off-target, the failure is upstream. If the retrieved content is correct but the model still answers badly, the issue is usually in generation, prompting, temperature settings, model routing, or post-processing.

For operational review, security and AI teams should look at:

• Retrieval source and timestamp, including whether the index was current.
• Top-k results, ranking scores, and whether the right passage was surfaced.
• Prompt assembly, including truncation, template errors, and context loss.
• Model identity, version, and decoding settings used for the response.
• Post-generation filters that may have rewritten or suppressed the output.

This is where the OWASP Agentic AI Top 10 and the NIST AI Risk Management Framework help frame the control problem. OWASP emphasises failures in orchestration, tool use, and trust boundaries, while NIST focuses on mapping risks to measurable governance and monitoring practices. If the system uses a retriever backed by a vector database, the retrieval path should be evaluated like any other production dependency: freshness, access control, and determinism matter.

When agentic features are present, the distinction becomes more important because a retrieval miss can trigger a tool call, a fallback path, or an unsafe reasoning loop. That makes end-to-end tracing essential, including the prompt actually sent to the model rather than the prompt template on disk. These controls tend to break down when retrieval is distributed across multiple indexes and fallback models because evidence gets fragmented across services.

Common Variations and Edge Cases

Tighter tracing and logging often increases storage, privacy, and engineering overhead, requiring organisations to balance diagnostic clarity against data minimisation. That tradeoff matters because the most useful evidence for root-cause analysis can also contain sensitive prompts, proprietary documents, or user data. Best practice is evolving, but current guidance suggests redacting where possible while preserving enough structure to reconstruct the failure path.

Some cases do not fit a simple retrieval-versus-generation split. A retrieved passage may be correct but outdated, which is technically a data freshness issue even though the trace looks healthy. A model may answer correctly but cite the wrong source, which is often a generation or attribution failure. In agentic systems, the model may also be acting on a tool result that is itself wrong, so the true failure may sit in the tool chain rather than the retriever or the LLM. The MITRE ATLAS adversarial AI threat matrix is relevant where adversarial manipulation of prompts, inputs, or context can distort both retrieval and generation outcomes.

Where regulated or high-risk deployments are involved, use the CSA MAESTRO agentic AI threat modeling framework alongside trace review to capture orchestration risk, not just model error. There is no universal standard for this yet, but teams should document whether the failure was caused by bad retrieval, bad prompt construction, bad model selection, or post-processing. That classification is the difference between fixing the index and endlessly retraining a model that was never given the right context.

Related resources from NHI Mgmt Group

• How do you know whether an LLM evaluation is actually trustworthy?
• How do you know whether an AI-driven investigation workflow is actually trustworthy?
• How do you know whether SCIM is actually supporting lifecycle governance?
• How do you know whether federal ICAM offboarding is actually working?