What are the signs that an R-squared score may be misleading?

When a High R-squared Does Not Mean the Model Is Good

An R-squared score is easy to overread because it measures how much variance the model explains, not whether it explains it for the right reasons. A model can look strong on this metric and still be poorly specified, overfit, or weak on predictive error. The first warning sign is a good-looking score that does not translate into better out-of-sample performance.

Non-linearity is a common trap. If the relationship between inputs and outcome curves, bends, or depends on interactions, a linear model may produce a respectable R-squared while still missing the structure that matters. In that case, the score reflects partial fit rather than real model adequacy.

Small samples can make the number look more stable than it is. With limited data, a few points can disproportionately move the fit, so the metric may swing upward even when the model would not generalise. That is why a high score on a thin dataset deserves skepticism rather than celebration.

Why Irrelevant Features and Overfitting Distort the Score

R-squared can also rise when the model absorbs noise instead of signal. Adding irrelevant variables may inflate the score even if those features do not improve decision quality, especially when the model is flexible enough to chase accidental patterns. A higher score in that setting is often a sign of complexity, not insight.

Overfitting is the other major warning. When the model fits the training data too closely, R-squared may look impressive while the model performs poorly on new data. The practical check is whether the metric stays credible on validation data and whether the prediction errors remain acceptable where the model will actually be used.

Just as important, a low R-squared is not automatically a failure. Some useful models explain only a modest share of variance because the outcome is inherently noisy or because important drivers are unavailable. A low score becomes concerning when it accompanies clear underfitting, missing terms, or residual patterns that show the model has left systematic structure behind.

What to Compare Before Trusting the Metric

The safest interpretation comes from comparing R-squared with error metrics and diagnostics, not from using it in isolation. If MAE or RMSE remain weak while R-squared looks strong, the model may be getting the broad trend right but still missing the scale of individual errors. Residual plots, out-of-sample checks, and sensitivity to feature changes often reveal more than the headline score alone.

Model context matters too. A score that is acceptable for one business or scientific use may be useless for another if the cost of error is asymmetric or if the model drives operational decisions. The right question is not whether the score is high, but whether the model is stable, interpretable enough for its use case, and accurate where errors matter most.

Practitioner Guidance

What to verify: Check whether the score is supported by validation results, residual behaviour, and error metrics on held-out data. If R-squared improves but MAE or RMSE do not, treat that as a warning that the model is fitting variance without improving usefulness.

Decision rule: If the model is small-sample, highly flexible, or full of added features, assume the score may be optimistic until out-of-sample performance confirms it. If the outcome is non-linear or the residuals show structure, revisit the model form before trusting the number.

Practitioner takeaway: R-squared is a summary statistic, not a quality guarantee, so the real test is whether it agrees with prediction error, residual pattern, and generalisation.

Related resources from NHI Mgmt Group

• What are the signs that an LLM is producing unreliable or misleading outputs?
• What are the signs that passive DNS data is incomplete or misleading?
• What are the signs that a model’s ROC performance is weaker than the AUC score suggests?
• What are the signs that a model may have an unrecognised or misleading genealogy?