Common signs include low hit rate, poor context relevance, weak nDCG, and precision that drops as more chunks are added. If the model receives relevant documents but still answers badly, retrieval may be noisy or too broad. Teams should use thresholds and monitors to identify which metric is failing, then test chunk size, reranking, and embedding choices against an eval benchmark.
What underperformance looks like in practice
Retrieval usually fails before the model ever gets to “reasoning.” The clearest signal is a mismatch between what your retriever returns and what the answer needs: relevant documents are missing, the top results are only loosely related, or the best evidence is buried so far down the list that the generator never uses it effectively.
Operationally, that shows up as high variance in answer quality for similar queries, frequent “almost right” responses, and answers that sound fluent but cite the wrong facts. When teams add more chunks and quality gets worse, the issue is often not model weakness, but retrieval noise, weak ranking, or over-broad chunking.
A useful way to frame the problem is that retrieval quality is not one metric. Low hit rate suggests the right material is not being surfaced at all, while weak relevance or falling precision suggests the system is surfacing too much distractor content. If you have an eval set, compare failures by query type so you can tell whether the issue is recall, ranking, or chunk granularity.
One practical data point is that NHI Mgmt Group’s Ultimate Guide to Non-Human Identities notes that 5.7% of organisations have full visibility into their service accounts, a reminder that poor visibility often looks like poor retrieval: the system cannot surface what it cannot consistently index, normalise, or connect.
Which metric is failing, and why that matters
Hit rate, precision, nDCG, and context relevance each describe a different failure mode, so a single “retrieval score” is rarely enough. If hit rate is low, the relevant source is not being retrieved. If precision drops as you add more chunks, the retriever may be broadening the candidate set faster than the ranker can prune it. If nDCG is weak, the right evidence may exist but not be ranked near the top.
The most common mistake is to tune the generation prompt when the real problem is candidate quality. If relevant documents are present but the answer is still wrong, the model is often receiving noisy context, conflicting passages, or partial evidence split across chunks. In that case, retrieval is underperforming even if the final answer sometimes looks acceptable.
Chunk size is especially important because retrieval can degrade in both directions. Chunks that are too small can lose surrounding context and create false negatives. Chunks that are too large can dilute the signal and bury the answer-bearing passage. Reranking and embedding choice matter because they change whether semantically similar but operationally irrelevant passages outrank the true evidence.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 address the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST AI RMF set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV-1 — Govern | Retrieval metric monitoring supports governance and accountability for LLM application quality. |
| Recommendation — Establish ownership and monitoring for retrieval quality metrics across the LLM workflow. | ||
| CIS Controls v8 | 8 — Audit Log Management | Metric monitoring and thresholding rely on observable telemetry and reviewable evidence. |
| Recommendation — Log retrieval outcomes and alert on abnormal drops in hit rate or precision. | ||
| OWASP Agentic AI Top 10 | A2 — Prompt Injection and Output Manipulation | Retrieval noise and poor context selection can amplify agentic answer errors under adversarial inputs. |
| Recommendation — Test retrieval against prompt-injection and context-poisoning scenarios in evals. | ||
| NIST AI RMF | GOVERN — Govern | LLM retrieval quality needs governance, measurement, and risk ownership within AI operations. |
| Recommendation — Define quality thresholds and review retrieval metrics as part of AI governance. | ||
Practitioner Guidance
What to verify: Separate retrieval evaluation from answer evaluation. You want a benchmark that tells you whether the right passage was retrieved, whether it was ranked high enough, and whether the context set stayed tight enough for the model to use.
Decision rule: If answer quality improves when you hand the model the correct passage directly, the retrieval layer is the likely bottleneck. If direct-passage performance is still poor, the issue may sit in the prompt, synthesis pattern, or source quality rather than retrieval alone.
Common mistake: Adding more chunks to “cover everything” without watching precision at the same time. More context can raise recall while quietly lowering usefulness, which is exactly how retrieval metrics can look busy while practical performance gets worse.
What to measure: Track metric behavior by query class, not just overall averages. A system can look healthy on easy factual lookups and still fail on multi-hop, long-tail, or ambiguous questions, so the real signal is which query patterns degrade first.
Practitioner takeaway: Treat retrieval as a ranking and selection problem first. The fastest path to improvement is usually identifying whether the failure is missing evidence, weak ordering, or excessive noise, then testing chunking, reranking, and embedding changes against a fixed eval set.
Related resources from NHI Mgmt Group
- What are the signs that an LLM application is being manipulated by prompt injection?
- What are the signs that an LLM application is failing its security controls?
- What are the signs that an LLM application is mishandling outputs and creating downstream security exposure?
- Why do LLM guardrails fail in ways that traditional application controls do not?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 17, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
