Topic starter
02/08/2026 12:27 pm
TL;DR: LLM feature quality should be measured with behavior, feedback, and laboratory metrics, and lab-based evals are the most reliable way to compare prompts, models, and hyperparameters without waiting on production noise, according to Semgrep. The practical shift is to treat prompt testing as reproducible infrastructure, not a one-off experiment.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “Does your LLM thing work? (& how we use promptfoo)”.
Key questions
Q: How should teams evaluate LLM features when outputs are non-deterministic?
A: Use a layered approach.
Q: Why do production metrics often fail to show whether an AI feature works?
A: Production is noisy because prompts, models, user behaviour, and release timing all change together.
Q: What breaks when prompt variables are not immutable and serializable?
A: You lose reproducibility.
Practitioner guidance
- Define separate eval targets for behavior, feedback, and laboratory metrics Use behavior metrics only when the AI feature has a measurable business outcome, use feedback for user-rated quality, and use laboratory evals for reproducible prompt and model comparison.
- Snapshot prompt inputs before rendering Store template variables in an immutable, serializable form so each test run can be replayed later with a different prompt, model, or temperature setting.
- Move conditional rendering into the templating layer Keep if conditions and loops out of application code so the prompt text used in evals matches the text used in production execution.
Bottom line: LLM quality cannot be judged reliably from production noise alone because prompts, models, and user conditions all change at once.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 1:26 pm
Lab metrics are the only dependable control surface for LLM change management. Production usage is too entangled with user behaviour, rollout timing, and workload noise to tell you whether a prompt or model change helped. Laboratory evals create a repeatable comparison point, which is exactly what governance teams need when they are deciding whether an AI feature is safe to ship.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: Should teams use LLMs to grade LLM outputs?
A: Yes, but only where the task allows multiple acceptable answers and the scoring criteria are explicit. Model-graded evaluation works best when it is anchored to a human gold standard and supplemented with style or policy checks. It should complement, not replace, carefully designed test cases and human review.
👉 Read our full editorial: LLM product quality depends on lab metrics, not production guesswork