What data quality failures most often break machine learning projects?

Why Data Quality Failures Break ML Before Model Tuning Does

Machine learning projects usually fail on data before they fail on algorithms. Duplicates, label conflicts, impossible values, and missingness can all make a dataset look usable while quietly undermining training and evaluation. That matters because model tuning cannot repair unreliable ground truth or unstable feature distributions. NIST SP 800-53 Rev 5 Security and Privacy Controls is useful here because it reinforces the broader control expectation that data handling needs integrity checks, not just storage and access controls. In practice, many teams discover these failures only after a model has already been promoted from a dataset that looked clean on the surface.

How These Failures Distort Training, Validation, and Deployment

Each failure mode breaks a different part of the ML pipeline. Duplicated records can overweight certain examples and make the model think a pattern is more common than it really is. Conflicting labels create contradictory supervision, which is especially damaging in classification tasks because the model is effectively trained to memorise disagreement. Impossible feature values, such as negative ages or out-of-range timestamps, can reveal upstream parsing errors or unit mismatches, and those errors often have a wider blast radius than a single column.

Excessive missingness is more subtle. Missing values are not always a problem, but when they cluster in specific classes, sources, or time periods, they can bias both training and evaluation. Quasi-constant features are another common trap: they survive schema checks but contribute little predictive signal and can add noise, storage cost, and operational complexity. They are sometimes kept because they are easy to collect, not because they are useful.

A practical review process usually separates these checks into three layers: record-level integrity, feature-level plausibility, and target-label consistency. That sequence helps teams catch structural issues before they become model issues. It also keeps validation honest, because a test set contaminated by the same defects as the training set may make a poor model look acceptable. Where this guidance breaks down is in highly specialised domains where rare values are genuinely expected and must be validated against domain rules rather than generic thresholds.

• Check for duplicate identifiers, repeated event signatures, and repeated rows that differ only in non-material fields.
• Compare labels across source systems when the same entity or event can be reclassified downstream.
• Validate ranges, formats, units, and time ordering before feature engineering.
• Measure missingness by segment, not just overall, to catch biased absence patterns.
• Remove or justify near-constant fields that add little signal but increase noise.

Edge Cases That Need Domain Rules, Not Generic Cleaning

Tighter data validation often increases pipeline overhead, requiring teams to balance error detection against the speed of ingestion and experimentation.

Not every repeated, missing, or constant-looking value is a defect. In operational data, some duplicates reflect legitimate retries, some missing values are meaningful by design, and some low-variance fields become important only in combination with other features. That is why practitioners should treat generic data-quality thresholds as a starting point, not a final judgment. The real question is whether the pattern is explainable in the source process and stable across time, sources, and populations.

There is also a consensus gap around how aggressively to correct imperfect data versus documenting and modelling around it. For some use cases, imputation or deduplication is appropriate. For others, especially regulated or safety-sensitive applications, silent repair can hide evidence of process failure. When the defect is tied to the label itself, the safest response is often to fix the upstream annotation workflow rather than to patch the dataset after the fact. In practice, the most reliable teams treat data quality as an operational control, not a one-time cleansing task.

Practitioner Guidance

What to prioritise: Start with defects that can silently distort both training and evaluation, especially duplicate records and label conflicts. Those two failure modes usually create more model risk than cosmetic feature issues because they affect the target signal itself.

What to verify: Verify that every flagged issue is explainable against source-system behaviour before you suppress it. A dataset is not trustworthy just because it passes schema validation; it is trustworthy when its anomalies are understood, bounded, and consistently handled.

Decision rule: If a data defect changes the meaning of the label or the representativeness of the validation set, treat it as a pipeline issue rather than a model-tuning issue. If it only affects a small, well-understood subset, document the exception and measure whether it changes performance materially.

What practitioners underestimate: Teams often underestimate how many ML failures come from inconsistent human annotation and source-system drift, not from exotic edge cases. The useful discipline is to treat data-quality review as part of model governance, because the model can only learn from the reliability of the inputs it is given.

Practitioner takeaway: The fastest way to improve ML reliability is usually not a better algorithm, but a stricter definition of what counts as valid training data.

Related resources from NHI Mgmt Group

• What do organisations get wrong about data quality in machine learning pipelines?
• Why do synthetic data pipelines often fail to improve model quality?
• Why do data quality failures keep surfacing late in organisations?
• How should teams govern archived data quality failures without creating another uncontrolled data store?