They fail because they compress many behaviours into one number. That makes them useful summaries but weak governance evidence, since they cannot show whether performance is stable across cohorts, whether edge cases are being missed, or whether the model is consistently wrong for a specific group.
Why a Single Metric Misses the Quality Story
Accuracy and precision are easy to compute, which is exactly why teams reach for them first. The problem is that they answer a narrow question: how often did the model get the label right, and how concentrated were the positive predictions? They do not tell you whether the model behaves reliably across different users, contexts, thresholds, or error costs.
That limitation matters because model quality is multidimensional. A system can look strong on a headline metric while still failing in the places that matter most, such as rare but high-impact cases, skewed populations, or workloads where false negatives are more damaging than false positives. Good governance needs a fuller picture than one aggregate score.
In practice, this is why quality review usually expands beyond a single summary number into calibration, class balance, cohort comparison, and error analysis. For readers working on governance or assurance, the same logic applies whether the model is statistical, operational, or embedded in a larger decision workflow. The metric is a signal, not the conclusion.
What Accuracy and Precision Hide
Accuracy can be misleading when class distributions are uneven. A model can achieve a high score by mostly predicting the majority class, even if it is poor at identifying the minority class that practitioners actually care about. Precision has a different blind spot: it ignores how many relevant cases the model misses, so it can look strong while recall remains weak.
That is why two models with the same accuracy or precision can still be very different operationally. One may be stable across cohorts and edge cases, while another may concentrate errors in a particular segment or fail only under unusual conditions. A single metric collapses those distinctions and hides whether the model is reliable or merely averaged out.
NIST AI Risk Management Framework is useful here because it pushes teams to evaluate model performance in context, not as an isolated score. That framing is especially important when the model influences decisions with different costs for different error types.
How to Judge Quality More Responsibly
Practitioners should treat accuracy and precision as starting points, then ask what kinds of mistakes the model makes and who bears them. The useful questions are usually cohort-based: does performance shift by segment, does calibration drift over time, and do rare classes or edge cases get systematically ignored?
It also helps to separate ranking quality from decision quality. A model may be good at ordering predictions but still require a threshold choice, manual review step, or policy guardrail before it becomes operationally acceptable. In other words, the metric can support selection, but it cannot by itself justify deployment.
NIST SP 800-53 Rev 5 Security and Privacy Controls is relevant because governance depends on documented measurement, monitoring, and review, not just a one-time performance claim. For model programmes, that means retaining evidence of how quality was assessed, what subgroups were checked, and which failure modes were accepted or mitigated.
Risk and Threat Considerations
When accuracy and precision are used as the only quality evidence, the main risk is false confidence. That can let a model ship with hidden cohort disparities, threshold fragility, or a failure pattern that only appears in edge conditions or under distribution shift.
Failure mechanism: A single aggregate metric averages away the error structure, so the model can appear acceptable even when it is consistently wrong for a specific group, underclassified on rare events, or poorly calibrated for the decisions it supports.
Impact: Teams may under-detect harm, overstate readiness, and miss the need for guardrails, recalibration, or human review in the exact places where the model is most likely to fail.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST AI RMF and NIST SP 800-53 Rev 5 set the technical controls, while ISO/IEC 42001:2023 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST AI RMF | Measurement and management of AI risk | Model quality needs context-sensitive evaluation beyond a single metric. |
| Recommendation — Assess performance, fairness, and robustness together before approving deployment. | ||
| NIST SP 800-53 Rev 5 | CA-7 — Continuous Monitoring | Model quality requires ongoing monitoring, not a one-time score. |
| AU-6 — Audit Review, Analysis, and Reporting | Quality claims need reviewable evidence of errors and subgroup performance. | |
| Recommendation — Monitor model behaviour over time and flag material drift or degradation. Retain and review evaluation evidence that shows how the model was tested. | ||
| ISO/IEC 42001:2023 | 8.2 — AI risk treatment and controls | AI governance must evaluate whether performance evidence is sufficient for use. |
| Recommendation — Define control requirements that go beyond a single aggregate metric. | ||
Practitioner Guidance
What to verify: Check cohort-level results, confusion-matrix behaviour, calibration, and performance on rare or high-cost cases before accepting a summary metric as evidence of quality. If the business impact differs by error type, the evaluation must show that the model remains acceptable under that asymmetry.
What good looks like: A defensible model review can explain where the model performs well, where it degrades, what was done about weak segments, and why the residual risk is acceptable for the intended use. If that explanation cannot be produced, the metric is too thin to support governance.
Practitioner takeaway: Use accuracy and precision as indicators, but never as the final word, because model quality is proven by stable, decision-relevant behaviour across the cases that matter most.