How should data teams handle data quality checks when source systems do not support native processing?

Why This Matters for Security Teams

When source systems cannot run native data quality checks, the risk is not just technical incompatibility. It is delayed detection, inconsistent validation, and gaps that quietly let bad records move downstream. In regulated and operationally sensitive pipelines, that means exceptions may be discovered only after reports, models, or alerts have already consumed them. NHI Management Group research shows how quickly weak control points become exposure points, especially when identities and credentials are widely distributed and poorly governed, as reflected in the Ultimate Guide to NHIs - Key Research and Survey Results.

Security and data teams should treat this as an execution-location problem, not a reason to weaken standards. Controls still need to verify completeness, format, freshness, lineage, and anomaly thresholds, but they may need to run in a separate engine, orchestration layer, or downstream validation service. That aligns with the control intent behind NIST SP 800-53 Rev 5 Security and Privacy Controls, which expects organisations to enforce control objectives even when implementation varies by system.

In practice, many data teams discover their weakest quality checks only after a source outage, schema drift event, or customer-facing reconciliation failure has already occurred.

How It Works in Practice

The practical model is to separate the quality policy from the source execution mechanism. The source system remains the system of record, but validation runs where the data can be safely observed, transformed, or sampled. That may be a streaming engine, ingestion pipeline, warehouse task, or control plane service. The core requirement is that the check executes consistently, produces evidence, and is tied to the dataset or event batch it validated.

A workable design usually includes three layers. First, define the check once as a reusable rule: row counts, null thresholds, referential integrity, schema compatibility, freshness, or domain constraints. Second, route the rule to an execution target that can actually run it, such as an orchestration job or a downstream processing engine. Third, capture results, timestamps, and exception handling so failures are auditable. This is the same governance mindset behind the Ultimate Guide to NHIs - Lifecycle Processes for Managing NHIs, where control is maintained across the full operational path rather than assuming every system can enforce every rule internally.

• Use the source for extraction, not necessarily for validation.
• Run checks in the nearest trusted execution environment that has the needed data access.
• Fail fast for critical constraints, but quarantine non-critical anomalies for review.
• Record lineage so every failed check can be traced back to the source, batch, and rule version.
• Apply the same policy across engines, even if the runtime differs.

For teams building the control plane, NIST SP 800-53 Rev 5 Security and Privacy Controls is a useful reference for evidence, monitoring, and integrity expectations. These controls tend to break down when validation is pushed too far downstream in high-volume event streams because bad data can cascade before the failure is detected.

Common Variations and Edge Cases

Tighter validation often increases pipeline complexity, so organisations have to balance stronger assurance against latency, cost, and operational overhead. That tradeoff is especially visible when source systems are legacy platforms, vendor-managed applications, or event streams with limited read guarantees.

Current guidance suggests a few patterns, but there is no universal standard for this yet. For static batch sources, downstream warehouse validation is often sufficient. For near-real-time feeds, checks may need to run in the ingestion layer or message bus to stop corruption early. For highly restricted systems, teams sometimes rely on sampling, reconciliation, and compensating controls rather than full in-source validation. The important point is to be explicit about where the control executes and what risk remains unaddressed.

NHIMG research shows why this matters operationally: weak handling of credentials, access paths, and validation boundaries often leads to broader exposure once a control gap exists, as seen in the Ultimate Guide to NHIs - Key Research and Survey Results and the Schneider Electric credentials breach. In edge cases where the source cannot be read safely at all, teams may need to rely on producer attestations, contractual controls, or external reconciliation until a better execution path is available.

Related resources from NHI Mgmt Group

• How should security teams handle encrypted metadata when multiple people and systems need to use the same credential across different applications?
• How should security teams handle device inventory when procurement, shipping, and onboarding happen in different systems?
• How should security teams govern identities when employee data is split across identity and HR systems?
• How should healthcare organisations secure sensitive clinical files and credentials when data sharing spans multiple teams and systems?