Overfitting

Expanded Definition

Overfitting is a model quality problem that appears when a machine learning system captures noise, outliers, or incidental patterns from its training set instead of learning the underlying relationship it is meant to generalise. In security and AI operations, that matters because a model can look accurate during development while becoming brittle, overconfident, or unexpectedly revealing once exposed to new data.

For glossary purposes, overfitting is not the same as simple poor performance. A badly tuned model can be inaccurate for many reasons, but an overfit model specifically performs well on the data it has already seen and degrades when real-world inputs differ. That distinction is important in machine learning risk reviews, where teams need to separate generalisation failure from data leakage, label noise, or underfitting. The concept is closely related to memorisation, and in privacy-sensitive settings it can increase membership inference risk because the model may behave differently for records that were part of training. The most common misapplication is treating every high-confidence error as overfitting, which occurs when the actual issue is distribution shift or weak feature design rather than excessive training-set fit.

Examples and Use Cases

Implementing overfitting controls rigorously often introduces a tradeoff between model fit and simplicity, requiring organisations to weigh short-term benchmark gains against long-term generalisation and privacy risk.

• A fraud detection model scores extremely well on historical cases but misses new attack patterns because it learned transaction quirks from one quarter of data.
• A medical triage model performs strongly in validation splits drawn from the same hospital, yet degrades when deployed in a different clinical environment with different coding practices.
• A recommendation model reproduces training-set preferences so closely that rare user profiles are treated as near-duplicates rather than distinct behaviour patterns.
• An LLM fine-tuned on a narrow internal corpus begins to echo specific phrasing from the data, which can expose sensitive examples and reduce output diversity.
• Security teams use holdout testing, regularisation, and data augmentation to check whether improvements reflect genuine learning rather than accidental fit. For governance context on managing AI system risk, see the NIST Cybersecurity Framework 2.0.

Why It Matters for Security Teams

Overfitting matters because it creates false confidence in model behaviour. Teams may approve a model that appears stable in testing but becomes unreliable in production, where inputs are messier, more adversarial, and less representative of the training set. In AI security, that gap can amplify downstream harms such as misclassification, over-personalised responses, weak anomaly detection, or leakage of sensitive training data through model outputs.

For identity and access use cases, overfitting is especially relevant when models support fraud detection, verification, or risk scoring. If the model has memorised training artefacts, it may overvalue signals that never generalise, causing false positives for legitimate users or false negatives for attacker patterns. That is why model evaluation should include data separation, stress testing, and monitoring for drift after deployment. NIST’s governance approach is useful here because it encourages repeatable risk management rather than one-time accuracy checks. Organisations typically encounter the operational cost of overfitting only after a model fails on live traffic, at which point retraining, retriage, and control redesign become unavoidable.