Shrinking the telemetry stack of needles in security operations

At a glance

What this is: This is an analysis of telemetry sprawl in security operations, arguing that the real problem is not only detection volume but misaligned data collection and storage strategy.

Why it matters: It matters because IAM, NHI, and broader security teams need to align log intake, retention, and detection use cases so analysts are not buried under data that never informs response.

Context

Security operations often treat more telemetry as inherently better, but that assumption breaks down when collection volume exceeds the actual detection value of the data. The issue is not whether logs are useful in principle, but whether every source needs to land in expensive, high-speed analytic storage to support day-to-day investigations.

This becomes a governance problem when compliance, visibility, and caution drive indiscriminate ingestion. For identity-heavy environments, the same logic applies to authentication and privilege data, where analysts need the right signals at the right speed rather than every possible event in the hottest tier.

Key questions

Q: How should security teams decide which telemetry belongs in hot storage?

A: Teams should keep only the telemetry that supports a real detection, investigation, or response decision in hot storage. If a log source does not change alerting, enrichment, or triage in time, it belongs in cheaper retention tiers. The goal is to preserve visibility while reducing the cost and search burden of unnecessary high-speed data.

Q: Why does collecting everything make security operations harder?

A: Collecting everything expands search scope, increases storage cost, and slows investigations because analysts must query through far more low-value data. That does not improve security if the extra telemetry never feeds a rule, model, or case. The operating model becomes less effective when volume replaces relevance as the collection principle.

Q: What are the signs that telemetry collection is out of control?

A: Common signs include soaring event volume, analysts complaining about slow searches, and large amounts of system telemetry sitting in premium analytics without a clear detection use case. Another indicator is when teams cannot explain why a source is ingested beyond vague visibility goals or compliance caution.

Q: How do security teams balance compliance retention with detection needs?

A: Use tiered storage. Keep only time-sensitive security signals in the fastest analytics layer, then preserve the rest in lower-cost storage that still satisfies retention, audit, and forensic requirements. That approach maintains accountability without forcing every event through the same expensive operational path.

Technical breakdown

Why telemetry volume outgrows analytic value

Telemetry stacks grow when teams equate completeness with security. In practice, many event types are valuable for retention, audit, or forensics but add little to real-time detection. Security operations platforms are built to optimise search, correlation, and alerting, so flooding them with low-value data increases cost and latency without improving decision quality. The architecture problem is not volume alone, but misclassification of data purpose: detection data belongs in high-performance analytics, while contextual and archival data can live elsewhere.

Practical implication: Classify each log source by detection value before routing it into the primary SIEM or analytics tier.

How detection logic should drive log collection

A detection-led collection strategy starts from the use case, not the source system. If the team is hunting authentication abuse, privilege escalation, or anomalous access patterns, then logs tied to those behaviours deserve priority. High-volume system telemetry such as CPU or memory events may support troubleshooting, but it rarely supports security detection. Mapping detections to sources exposes a simpler architecture: only the events that can trigger rules, train models, or corroborate an incident need premium treatment.

Practical implication: Tie every ingested source to a named detection or investigative use case, and drop sources that do not support one.

Why hot storage should not hold every event

Hot storage is expensive because it is optimised for speed, not bulk retention. When organisations keep everything in the fastest tier, they pay for low-value data twice: once to ingest it and again to search it. The better pattern is tiered storage, where high-priority signals remain searchable in near real time and lower-value records move to cheaper object storage for compliance, longer retention, or later forensics. This preserves visibility while reducing analyst friction.

Practical implication: Move non-detection telemetry to lower-cost storage tiers and reserve hot analytics for events that change security decisions.

NHI Mgmt Group analysis

Telemetry sprawl is a governance failure before it is a tooling problem. Security teams often chase broader collection because it feels safer, but the outcome is an oversized analytic stack that slows investigations and obscures the signals that matter. The core issue is not the absence of data but the absence of collection discipline tied to a clear detection model. Practitioners should treat telemetry scope as a control boundary, not a default setting.

Detection relevance, not source completeness, should determine storage tier. The article makes the case that much of what lands in expensive real-time analytics never drives an alert or enriches an investigation. That is a control design problem, not just a cost problem. For security programmes, the right question is whether a data source changes a decision in time, not whether it exists in case someone may need it later.

Identity and privilege telemetry is the right place to be selective and precise. Authentication events, privilege changes, and access patterns are the data most likely to support identity-driven detections, which means they deserve priority over bulk system noise. This is especially important where NHI and human IAM overlap, because access events are valuable only when they map to a governed detection or response need. Security teams should narrow the stack around the access behaviours that actually change risk.

The named concept here is telemetry stack reduction. That means shrinking the volume of data in high-performance security analytics without reducing security coverage. It is a practical response to the log lemming effect, where teams keep ingesting everything because they fear missing something. The stronger control is selective ingestion backed by explicit detection mapping, and practitioners should build their telemetry programme around that principle.

What this signals

Telemetry stack reduction: security programmes need to distinguish between data that drives live detection and data that only supports retention or forensics. When that distinction is absent, the SIEM becomes a storage problem disguised as a monitoring strategy.

For identity-heavy environments, the most valuable telemetry is often the most selective. Authentication, privilege, and access events should remain tightly governed because they change security outcomes, while bulk operational noise should not be allowed to dominate the analytic tier.

For practitioners

• Define detection-backed ingestion rules Require each telemetry source to map to at least one named detection, investigation, or compliance purpose before it enters the primary analytics stack.
• Split hot and cold telemetry paths Route high-value security events into real-time analytics and move low-value system telemetry to cheaper object storage for retention and forensics.
• Prune low-value system noise Remove CPU, memory, and other operational logs from expensive security analytics when they do not support active detections.
• Review telemetry intake with analysts Ask analysts which sources they actually use during investigations, then remove or re-tier sources that never affect triage or response.

Key takeaways

• Security operations degrade when telemetry collection grows faster than the team's ability to use it.
• The core design mistake is treating every log source as if it needs premium real-time analytics.
• A tiered model that keeps detection-critical events hot and pushes the rest to cheaper storage improves both cost and investigation speed.

Key terms

• Telemetry Stack Reduction: Telemetry stack reduction is the practice of shrinking the amount of data sent to high-performance security analytics without losing meaningful detection coverage. It separates live detection needs from retention and forensic needs so teams can lower cost, speed up search, and reduce analyst fatigue.
• Detection-led Ingestion: Detection-led ingestion is a collection model where each log source is admitted to the analytics pipeline only if it supports a named security detection, investigation, or response use case. It replaces blanket collection with deliberate, evidence-based routing decisions.
• Hot Storage: Hot storage is the tier reserved for security data that must be available immediately for analysis, alerting, and triage. It is typically the most expensive tier, so teams should limit it to telemetry with the highest operational value and the strongest likelihood of supporting timely detections.
• Tiered Retention: Tiered retention is a storage strategy that places different data classes into different cost and performance tiers based on business and security need. It lets organisations keep low-value telemetry for compliance or forensics without forcing it through premium analytics infrastructure.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, identity lifecycle, and machine identity security. It helps security and IAM practitioners build stronger control models where access telemetry and identity risk intersect.