How should teams build self-improving LLM evaluations from real application data?

From traces to a golden set that actually reflects failure patterns

Self-improving LLM evaluations work best when the dataset is built from production traces first, because traces capture the prompts, edge cases, and task context that synthetic examples often miss. The most useful evaluation set is not a gallery of polished answers, but a curated sample of real failures, near misses, and ambiguous cases that show how the system behaves under actual usage.

That means the first design decision is sampling strategy. Teams should deliberately include routine successes, degraded outputs, safety-adjacent failures, and user journeys where the model had incomplete context, because those examples reveal where the evaluator needs sharper criteria. A golden dataset becomes valuable only when it captures the distribution of observed failure modes rather than a simplified benchmark version of the product.

When traces are sparse, synthetic cases are still useful, but they should fill coverage gaps rather than define the evaluation universe. The strongest practice is to label synthetic examples as synthetic, then keep them separate from trace-derived cases so you can see whether the evaluator generalises from real behavior or only performs well on constructed inputs. That distinction matters when you later decide whether a scoring rule is robust enough to trust.

How human review, criteria, and prompts should evolve together

Human review is the mechanism that turns raw examples into a usable evaluation system. Reviewers are not only judging outputs, they are clarifying what “good” means for the task, resolving borderline cases, and tightening the rubric so the evaluator can distinguish acceptable variation from true failure. Without that calibration loop, the dataset tends to encode reviewer preference rather than stable task criteria.

Prompting and criteria should be revised together because a better rubric can expose flaws in the evaluator prompt, and a better evaluator prompt can reveal where the rubric is underspecified. The practical goal is consistency, not perfection: if two competent reviewers would score the same case differently, the criterion still needs refinement before it becomes part of the golden set. Teams often underestimate how much score drift comes from vague labels rather than model quality itself.

Once the evaluator is running, it should be re-tested against new traces on a regular cadence and adjusted when its scores stop matching observed user impact. If the use case changes, the dataset should change too, because an evaluator trained on old workflows can look stable while quietly missing new failure modes. A self-improving loop only works when the evaluation standard is treated as a living artifact, not a one-time benchmark.

What good looks like when the evaluator becomes a product control

The best evaluator design is one that can explain its own disagreements with production reality. When a model output scores well but users still reject it, the team should be able to inspect the underlying example, understand why the rubric missed the issue, and decide whether to update the prompt, the criteria, or the dataset. That traceability is what makes the system self-improving rather than merely self-referential.

For teams operating at scale, the key signal is whether the golden dataset keeps pace with real traffic. If new failure patterns are appearing in logs but not in evaluation, the loop has broken. If the evaluator only improves on curated examples and not on fresh traces, it is likely overfitting to the review process. The objective is an evaluation set that remains faithful to live behavior while still being small enough to review deeply.

If the application depends on materially sensitive or high-stakes outputs, treat evaluation updates as a controlled change, not a casual prompt edit. The more the evaluator influences release decisions, the more important it becomes to keep versioned datasets, documented rubric changes, and a clear reason for each revision so that score movement can be explained later.

Practitioner Guidance

What to prioritise: Start with the failure modes that affect user trust, task completion, or downstream operational cost, then work backward to the examples that best expose those failures. A small set of well-chosen trace examples is usually more valuable than a broad synthetic benchmark with weak grounding.

What to verify: Check that reviewers can score borderline cases consistently before you freeze the rubric. If the same example produces unstable judgments, the issue is usually rubric ambiguity, not evaluator quality.

What practitioners underestimate: The evaluator itself needs maintenance. As product behavior, prompts, and traffic patterns change, the dataset and criteria must be revalidated or the scoring system will drift away from reality.

Practitioner takeaway: The most reliable self-improving evals are built like a feedback system, with real traces at the center, synthetic coverage only where needed, and human review used to keep the rubric aligned with actual failure behavior.

Related resources from NHI Mgmt Group

• What breaks when teams treat LLM output as if it were trustworthy application data?
• How should security teams build an application security program around real business risk instead of scan volume?
• What do security teams get wrong about self-service data APIs in real-time environments?
• Why do LLM applications create new data leakage risks for identity teams?