TL;DR: Security pipelines have moved beyond cost control: in cloud-native environments, data quality, classification, and early enrichment now determine whether telemetry is usable for detection, correlation, and response, according to Axoflow. The central claim is that distributed storage can work, but only if access, schema handling, and normalization are treated as security controls, not afterthoughts.
At a glance
What this is: This is an analysis of why security data quality, not storage location, has become the decisive control point for pipelines.
Why it matters: It matters because IAM, NHI, and broader SecOps teams depend on telemetry that is complete, correctly structured, and searchable before detection and response can work.
Context
Security data pipelines are no longer just transport and storage layers. The core problem is whether telemetry stays usable as it moves from source systems into detection and response workflows, especially when cloud-native environments create more distributed data paths and more schema variation than older centralised models.
The article argues that the pipeline now has to act as a governance layer for access, classification, normalization, and enrichment. That matters for identity and access programmes because machine and service telemetry, API activity, and privileged session data lose value quickly if the pipeline cannot preserve context and structure.
Key questions
A: Security teams should use a control plane approach that filters, samples, masks, enriches, and routes telemetry before it reaches every downstream tool. The goal is to reduce unnecessary volume early, keep costs predictable, and preserve enough signal for investigations. That works best when pipeline changes are governed, observable, and rolled out safely across environments.
Q: Why do malformed or truncated logs create more risk than just noisy analytics?
A: Because bad telemetry can erase visibility before detection ever begins. If records are truncated, duplicated, or mapped into the wrong schema, correlation logic cannot reliably reconstruct the event, and the security team loses evidence rather than simply getting an imperfect alert. The risk is operational blindness, not just inefficiency.
Q: What do security teams get wrong when they try to fix log quality inside the SIEM?
A: Teams often treat the SIEM as the place to clean up data, but that is usually slow, manual, and expensive. By the time logs arrive, ingestion costs are already locked in, routing choices are limited, and fixes become tightly coupled to one platform. That increases operational burden and vendor dependence.
Q: How do you know if a security data lake pipeline is actually working?
A: It is working when raw telemetry can be transformed into a stable schema, delivered into the target storage layout, catalogued correctly, and queried during real operational spikes. If schema changes, burst traffic, or destination-specific write rules regularly interrupt that flow, the pipeline is failing as a control.
Technical breakdown
Why centralised log storage is no longer the default
Traditional log architecture assumed that security data should be forced into one repository for correlation and retention. That model still has value for some use cases, but cloud-native systems, elastic storage, and federated search make it less necessary as a universal design. The real constraint is not storage volume alone. It is whether data remains accessible, searchable, and trustworthy wherever it lands. In this model, the pipeline becomes the control plane for movement and usability, while storage becomes one choice among several. Practical implication: design pipelines around retrieval and usability, not around a single storage destination.
Practical implication: design pipelines around retrieval and usability, not around a single storage destination.
How classification and normalization prevent blind spots
Security pipelines fail when they treat raw logs as if they were already analysis-ready. Classification means identifying source types, schemas, and field patterns so the pipeline can apply the right parsing and transformation automatically. Normalization means converting inconsistent records into a common structure that downstream tools can use reliably. Without those functions, malformed records, truncated messages, duplicate fields, and schema drift create visibility gaps that SIEM ingestion cannot repair later. This is especially important where telemetry spans multiple platforms and identity-related events need consistent fields for correlation. Practical implication: put schema awareness and automatic normalization before downstream analytics so bad records do not become permanent blind spots.
Practical implication: put schema awareness and automatic normalization before downstream analytics so bad records do not become permanent blind spots.
Why early enrichment changes the security value of telemetry
Enrichment is most effective when it happens close to the source, before important context disappears in transit or at ingestion. Adding asset metadata, location context, trusted network information, or schema mapping early gives downstream tools a richer event without depending on later reconstruction. That makes the pipeline more than a relay. It becomes the place where raw events are turned into security-ready records. For teams handling identity or workload telemetry, this matters because the value of an event often depends on the surrounding context, not just the event itself. Practical implication: enrich in the pipeline when the source context is still intact and the original event shape is still recoverable.
Practical implication: enrich in the pipeline when the source context is still intact and the original event shape is still recoverable.
NHI Mgmt Group analysis
Data quality has become a governance control, not an operational convenience. Once security telemetry is spread across cloud services, devices, and federated stores, the question is no longer where the data sits but whether it remains usable. Pipelines now determine whether evidence survives truncation, schema drift, and misclassification. The governance failure is to treat data quality as a downstream clean-up task instead of an upstream control.
Telemetry usability gap: this is the practical failure mode the article exposes. When logs are malformed, truncated, or mapped into the wrong schema, analytics teams do not get a noisy signal, they get no signal at all. That is a security architecture problem, because blind spots are created before detection tools ever see the event. Practitioners should treat pipeline validation and normalization as part of control design.
Early enrichment is an identity-adjacent control, not just a data engineering preference. When events are enriched in transit, the pipeline can preserve asset context, trusted network information, and other metadata that make access and privilege events interpretable. That matters wherever human and non-human identities generate high-volume telemetry, because later reconstruction is often too late. The practitioner conclusion is straightforward: preserve context before it is lost.
SIEM economics have distorted the control point for too long. If ingestion volume is rewarded more than data quality, the architecture incentivises collecting more broken telemetry instead of fewer usable events. That shifts responsibility to the pipeline layer, which is the only place that can see the raw record before transformation and repair it before the loss becomes permanent. Teams should re-evaluate where quality ownership actually belongs.
What this signals
Security teams should expect pipeline quality to become a board-level operational issue whenever telemetry is spread across cloud, endpoint, and service layers. The practical test is whether analysts can search and correlate the record immediately, not whether the record eventually lands in storage.
Telemetry usability gap: when classification and normalization happen too late, the organisation pays twice, first in ingestion cost and then in failed detection outcomes. The control objective is to preserve structure before transformation makes the original event harder to recover.
For practitioners
- Harden source-side schema validation Define expected event structures at collection time so malformed records, duplicate fields, and truncation are detected before they reach analytics systems.
- Move normalization into the pipeline Translate raw telemetry into consistent structures such as CEF, ECS, UDM, or OCSF before downstream indexing so correlation rules operate on stable fields.
- Treat enrichment as a collection-stage control Add asset metadata, location context, and trust-zone information while the event still retains its original source context.
- Measure data quality, not just ingestion volume Track parse success, field completeness, schema drift, and rejected-event rates so pipeline health reflects operational usability rather than raw throughput.
Key takeaways
- Security data pipelines now matter because the quality of the event, not the cost of the store, determines whether downstream detection can work.
- Malformed, truncated, or misclassified telemetry creates blind spots that analytics cannot reliably repair after ingestion.
- Teams need pipeline-stage normalization, enrichment, and schema validation if they want telemetry that is usable at the point of collection.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP API Security Top 10 addresses the attack and risk surface, while NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.DS-01 — Data-at-Rest Is Protected | Telemetry quality and preservation are central to the article's pipeline governance theme. |
| DE.CM-01 — The network and systems are monitored to detect potentially adverse events | The article is about making telemetry usable for detection and response. | |
| PR.PS-01 — Configuration Management | Manual pipeline configuration and schema drift are recurring failure points in the article. | |
| Recommendation — Apply PR.DS-01 to preserve security telemetry integrity from collection through storage and analysis. Validate telemetry quality so monitoring systems receive records that can actually support detection. Standardise pipeline configuration so parsing, routing, and normalization stay consistent across sources. | ||
| CIS Controls v8 | CIS-8 — Audit Log Management | The article focuses on keeping logs complete, parseable, and usable for security operations. |
| Recommendation — Manage audit logs so they remain complete, normalised, and available for investigation. | ||
| OWASP API Security Top 10 | API9 — Improper Inventory Management | Federated data access and distributed telemetry require disciplined inventory of sources and schemas. |
| Recommendation — Inventory telemetry sources and schemas so distributed data remains searchable and governed. | ||
Key terms
- Data Pipeline Normalization: The process of converting raw security telemetry into a consistent structure that downstream tools can search and correlate reliably. It reduces schema drift, malformed fields, and parser failures so events remain usable across storage, analytics, and response workflows.
- Classification: Classification is the practice of labelling data according to sensitivity, business impact, or regulatory need. It helps security teams distinguish PII, PHI, PCI, financial records, and intellectual property so protections can be applied consistently and proportionally across the environments where the data appears.
- Early Enrichment: Adding context to security events before they are indexed or handed to downstream analytics. This usually includes asset metadata, location, trust-zone, or identity context, and it matters because later processing often cannot reconstruct the original source conditions with equal accuracy.
- Telemetry Usability: The degree to which collected security data can be searched, understood, and acted on without extensive repair. Usability depends on completeness, structure, context, and timeliness, not simply on how much data has been ingested or stored.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the broader security programmes they operate.
Published by the NHIMG editorial team on August 19, 2026.
Updated on October 12, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org