Why is accuracy misleading on imbalanced datasets?

Why accuracy breaks down on imbalanced data

Accuracy measures overall correctness, but it does not tell you how the errors are distributed across classes. On imbalanced data, the majority class can dominate the score, so a model can appear strong while missing the cases you actually care about. That makes accuracy a poor standalone decision metric whenever the rare class is operationally important.

The core problem is that accuracy rewards “doing well” on the easy majority class more than it penalises failure on the minority class. If positives are rare, predicting the majority label most of the time can still produce a high score. In practice, that means a model may look statistically healthy while offering little real-world value.

The issue is not that accuracy is mathematically wrong. It is that it answers the wrong question for skewed data. If the business cost of missing a positive is high, the metric can hide the very failure mode that matters most.

What precision and recall reveal that accuracy hides

Precision and recall separate the two parts of positive-class performance that accuracy blends together. Precision asks how many predicted positives were actually correct. Recall asks how many of the true positives the model found. Together, they show whether the model is merely avoiding mistakes on the majority class or actually identifying the rare events.

That distinction matters because different failure patterns can produce the same accuracy. A model that rarely predicts the positive class may score well on accuracy, yet have poor recall. Another model may catch more positives but generate more false alarms, which can be acceptable or unacceptable depending on the use case.

For this reason, accuracy should usually be treated as a supporting metric rather than the decision metric. In skewed classification problems, class-specific metrics, confusion-matrix counts, and threshold analysis give a clearer picture of operational performance than a single aggregate percentage.

Why class imbalance changes the evaluation strategy

Imbalance changes what “good” means because the base rate is no longer representative of the outcome you care about. A model evaluated only on aggregate correctness can exploit the skew instead of learning the minority pattern. That is why practitioners often compare against a naïve baseline, such as always predicting the majority class, before trusting a reported accuracy figure.

It also changes threshold selection. A model can produce useful probability estimates but still fail at the default cutoff if that cutoff is not tuned for the minority class. In many real deployments, the better question is not “What is the highest accuracy?” but “What operating point best balances missed positives and false alarms for the task?”

When the dataset is heavily skewed, it is also worth checking whether the validation split preserves the same class distribution and whether the test set is large enough to measure minority performance reliably. Otherwise, the evaluation can become noisy enough that a seemingly strong score is little more than sampling luck.

Risk and Threat Considerations

Imbalanced datasets create a measurement risk because a deceptively high accuracy score can mask systematic failure on the rare but important class. In security, fraud, medical, and anomaly-detection settings, that can lead teams to deploy a model that looks stable in dashboards while missing the events that matter most.

Failure mechanism: The model optimises overall correctness against a skewed base rate, so majority-class predictions dominate the score while minority-class misses remain undercounted.

Impact: Missed positives can translate into undetected abuse, delayed response, or unsafe automation, while teams may falsely conclude the model is production-ready.

Practitioner Guidance

What to verify: Check the confusion matrix, precision, recall, and class prevalence before accepting any headline accuracy figure. If the minority class is the business-critical one, require evidence that the model performs adequately on that class, not just overall.

Decision rule: If the minority class matters more than the majority class, treat accuracy as a secondary reporting metric and choose a threshold based on the cost of false negatives versus false positives. If those costs are unclear, the model is not yet ready for an operational decision.

What practitioners underestimate: A model with “excellent” accuracy can still be functionally useless when the rare outcome is the one that drives risk, loss, or intervention. The right test is whether the model finds the cases that matter, not whether it preserves a flattering aggregate score.

Practitioner takeaway: On imbalanced data, accuracy is easiest to game and hardest to trust, so evaluation should start with minority-class detection quality and only then ask what the overall score means.

Related resources from NHI Mgmt Group

• Why can SHAP explanations become misleading in real datasets?
• How should teams choose F1 score instead of accuracy for imbalanced classification problems?
• Why does F1 score give a more honest signal than accuracy in skewed datasets?
• Why does binary cross entropy become less reliable on highly imbalanced datasets?