Notifications
Clear all
Topic starter
10/07/2026 9:16 pm
TL;DR: Measurable validation, not manual review, becomes the scalable control for complex data workflows, according to SecurityScorecard. Its scoring pipeline uses Great Expectations, DataHub, and Apache Airflow to standardize data quality checks across ingest, compute, database sync, and customer-facing release stages, replacing ad hoc validation with repeatable controls and observable results.
NHIMG editorial — based on content published by SecurityScorecard: data quality validation in Airflow pipelines
Questions worth separating out
Q: How should teams implement data quality checks in multi-stage pipelines?
A: Teams should attach checks to each meaningful boundary, not just to the start and end of the workflow.
Q: When do row-by-row validation checks become a bad design choice?
A: Row-by-row validation becomes a bad choice when the dataset is large enough that Python or application memory becomes the bottleneck.
Q: What do security and data teams get wrong about validation observability?
A: They often treat observability as a log viewer instead of a governance control.
Practitioner guidance
- Map validation to workflow boundaries Place checks before expensive compute, after transformation, before database sync, and at the final release gate so failures are caught where they matter most.
- Standardise validation definitions Use one repeatable format for schema checks, null checks, and range assertions so teams can reuse logic across repositories instead of maintaining one-off scripts.
- Centralise assertion results Publish pass or fail outcomes, timestamps, expected versus actual values, and run metadata into a shared observability layer so trends are visible across the programme.
What's in the full article
SecurityScorecard's full article covers the implementation detail this post intentionally leaves at a higher level:
- How the team structured Great Expectations checks across input, output, and database stages
- How DataHub stores assertions, timestamps, and expected versus actual values for review
- How the Airflow workflow pauses on failed validation and routes alerts to on-call teams
- How database-side SQL replaced slower Python validation for large business-rule checks
👉 Read SecurityScorecard's article on stage-by-stage data quality validation in Airflow →
Airflow data quality checks: what practitioners should change now?
Explore further
10/07/2026 9:39 pm
Validation debt is the real risk in complex data pipelines. Teams often accumulate ad hoc checks that work in one repository but fail to scale across orchestration, databases, and release stages. That creates hidden governance debt: controls exist, but they are inconsistent, hard to compare, and easy to bypass. The operational answer is not more testing in the abstract, but standardised validation that travels with the workflow.
A question worth separating out:
Q: Who is accountable when bad data reaches a downstream system?
A: Accountability should sit with the team that owns the final gate before the data is exposed to users or dependent systems. If that gate only reports errors and does not stop release, the control is incomplete. Effective governance requires clear owners, automatic blocking, and a visible record of why the pipeline was paused.
👉 Read our full editorial: Data quality in Airflow pipelines needs stage-by-stage validation
