A common mistake is treating eval data as static. Teams often collect examples once, then fail to keep the dataset current as the product changes. Another error is relying on manual file updates, which quickly becomes hard to maintain and collaborate on. A better approach is to instrument tracing early, review logs continuously, and promote interesting cases into eval datasets.
Why real interaction data becomes stale faster than teams expect
Eval datasets built from user interactions age quickly because the product does not stay still. Prompt patterns, tool behavior, retrieval sources, policy rules, and failure modes all shift as the system evolves, so a dataset that looked representative at collection time can become biased toward old traffic and old bugs.
The practical mistake is assuming that one pass of curation is enough. In reality, the value of real interaction data comes from how well it tracks the current product state, not from how authentic the original examples were. Teams that stop at collection often miss the cases that matter most after a release, a model swap, or a workflow change.
For teams building the review loop around real usage, the strongest pattern is to treat evaluation data as a living asset rather than a static archive. NHI Mgmt Group’s Ultimate Guide to NHIs is useful here as a reminder that operational data only stays useful when governance, rotation, and visibility are continuous, not one-time events. OWASP SAMM also supports the same discipline at the process level, because maturity depends on repeatable feedback loops rather than ad hoc cleanup.
- Refresh the dataset on a schedule tied to product change, not just calendar time.
- Track which product version, prompt flow, and tool stack generated each example.
- Retire examples that no longer reflect current behavior, even if they were once high quality.
Why manual file editing usually breaks the feedback loop
Teams often begin with spreadsheets or shared folders because they are easy to start, but those formats become brittle as soon as multiple people need to review, label, and promote examples. Manual file updates make provenance unclear, introduce merge conflicts, and turn dataset maintenance into a hidden coordination problem instead of a measurable process.
The deeper issue is not just convenience. When curation lives in files, there is no reliable way to answer basic operational questions such as who added a case, why it was promoted, whether it was duplicated, or whether it still matches the current policy surface. That weakens trust in the eval set and makes iteration slower, not faster.
Instrumented tracing solves the maintainability problem because it creates a structured pipeline from observed interaction to review to dataset entry. DeepSeek breach is a reminder that logs can contain extremely sensitive material and must be handled carefully, while still being valuable for detection and learning. For broader supply-chain discipline in how data enters the workflow, SLSA is a useful reference point for provenance-minded thinking, even when the assets are evaluation artifacts rather than build outputs.
What good practice looks like when you promote real cases into evals
The best teams do not treat every user interaction as eval material. They use traces and logs as the source pool, then promote only cases that reveal a meaningful edge condition, a repeated failure mode, or a representative success path that should stay in regression coverage. That keeps the dataset focused and avoids drowning signal in routine traffic.
The other key judgment is selection discipline. A case is worth promoting when it teaches you something durable about the system, not when it is merely unusual. If a sample is tied to a temporary experiment, a one-off outage, or a soon-to-change policy, it may belong in analysis notes, but not in the permanent eval set unless the underlying pattern is likely to recur.
Teams also need a clear ownership model for review and promotion. Real interaction data touches product quality, safety, and operational logging, so the process should define who can approve additions, what evidence must be retained, and when a case should trigger a new test category. 52 NHI Breaches Analysis is a useful analog for why repeated real-world failure patterns deserve structured capture, and NIST Cybersecurity Framework 2.0 is a solid baseline for organizing the governance, detection, and response responsibilities around that loop.
Practitioner takeaway: The strongest eval datasets are not the biggest ones, they are the ones that stay current, traceable, and selective enough to reflect how the product behaves now.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 provides the primary governance reference for this topic.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | GV — Govern | Promoting real cases needs ownership, policy, and lifecycle governance. |
| DE.AE — Anomalies and Events | Real interaction logs are the source of meaningful edge cases and anomalies. | |
| RS — Respond | Interesting cases from real usage should feed an operational response loop. | |
| Recommendation — Define ownership and governance for dataset updates and case promotion. Continuously review events and logs to identify cases worth promoting into evals. Use incident and review outcomes to update eval coverage and priorities. | ||
Related resources from NHI Mgmt Group
- What do security and governance teams get wrong about AI training datasets?
- What do security teams get wrong about user feedback on AI outputs?
- What do teams get wrong about comparing AI models in real-world applications?
- What do teams get wrong about building frictionless user experiences for modern SaaS customers?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 18, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
