Should teams keep the full telemetry stream outside the SIEM?

Why a split telemetry architecture makes sense

A full-stream-outside-the-SIEM design is usually about preserving detail, not avoiding detection. The SIEM remains the place for prioritized alerts, correlation, and analyst action, while the lake retains the raw, high-volume record needed for long-horizon search, compliance, and investigations that need more context than a normal retention tier can carry.

That separation matters when teams are dealing with noisy infrastructure, bursty cloud logs, or telemetry that is useful only after the fact. If every event must pass through the same expensive pipeline, teams often end up sampling, truncating, or normalizing too early, which can reduce evidence quality exactly when responders need it most. Keeping the source stream intact also gives investigators a better basis for reprocessing data as detections evolve.

A well-designed split does not mean “SIEM for some, storage for the rest” by accident. It means defining which events need fast alerting, which need durable retention, and which need both. The practical goal is to separate detection latency from evidence preservation, so the security program is not forced to trade fidelity for cost every time log volume spikes.

What stays in the SIEM and what stays in the lake

The SIEM should hold the subset of telemetry that supports active defense: high-risk security events, authentication anomalies, privileged actions, policy violations, and other signals that benefit from correlation and alerting. The lake should keep the wider stream, including verbose application logs, control-plane telemetry, and supporting context that analysts may need later when a smaller alert set is no longer enough.

This is most effective when the routing logic is intentional. A raw event may be too noisy for always-on SIEM storage, but still critical if it can later explain a timeline, confirm scope, or reconstruct attacker behavior. For that reason, teams should think in terms of use cases, not source systems. One source can feed both paths if the event class has operational value in both places.

Retention policy should follow that split. The SIEM is optimized for timeliness and workflow efficiency, while the lake is optimized for depth, replayability, and lower-cost storage. NIST Cybersecurity Framework 2.0 is useful here because it separates governance, detection, response, and recovery into distinct operational outcomes rather than forcing every control objective into one pipeline.

How to avoid creating blind spots or wasted spend

The main failure mode is assuming the lake can substitute for detection, or that the SIEM can substitute for evidence retention. Neither is true on its own. If the SIEM only receives a reduced view, teams can miss short-lived abuse patterns; if the lake is poorly indexed or hard to query, the organization may retain data it cannot actually use during an incident.

Another common mistake is pushing everything into the SIEM “just in case.” That often drives up ingest costs, throttling risk, and analyst fatigue, and it can dilute the value of the alert stream. A better pattern is to preserve the full stream separately, then apply explicit forwarding rules for the events that genuinely need immediate correlation. That approach is consistent with the idea of limiting the expensive path to the telemetry that earns it.

Telemetry architecture also becomes a governance issue when teams cannot explain why a given dataset is retained, where it is processed, or which workflow depends on it. NIST SP 800-53 Rev 5 Security and Privacy Controls is relevant because audit logging, access control, and retention-related controls all depend on being clear about what is collected, where it lives, and who can query it.

Practitioner Guidance

What to verify: Confirm that the SIEM receives the events required for alerting, while the lake preserves the raw fields needed for investigations, compliance queries, and reruns of detection logic. If either path drops fields, timestamps, or correlation keys, the split is not doing its job.

Decision rule: If an event must drive near-real-time alerting, forward it to the SIEM; if it is primarily evidence, context, or long-horizon hunting material, keep the complete record in the lake and index it for retrieval rather than forcing it through the alert pipeline.

Common mistake: Treating the lake as passive archive only. In practice, the lake should be queryable enough to support retrospective hunts and incident reconstruction, otherwise the organization has paid to store data it cannot operationalize.

Practitioner takeaway: The right question is not whether to keep the full stream outside the SIEM, but whether the organization has cleanly separated fast detection from durable evidence without making either layer incomplete.

Related resources from NHI Mgmt Group

• How should teams keep compliance data available while lowering SIEM spend?
• What do teams get wrong about filtering telemetry before the SIEM?
• What breaks when security teams keep telemetry and intelligence in separate stores?
• When should telemetry be retained outside the SIEM?