Shrinking the telemetry stack of needles in security operations

At a glance

What this is: This is an analysis of telemetry growth in security operations and the claim that most data collected for detection does not need to live in the high-cost analytic stack.

Why it matters: It matters because IAM, PAM, and NHI-relevant events are often buried inside oversized pipelines, so practitioners need to separate detection-critical identity signals from retention-only noise.

By the numbers:

• Back in the early 2000s, a large SIEM deployment was around 50,000 events per second, maybe five terabytes a day.
• Now, medium-sized environments operate at that scale, and the largest ones push a million events per second, which works out to roughly 100 terabytes a day.

👉 Read Abstract Security's analysis of shrinking the security telemetry stack

Context

Security operations teams often treat more telemetry as more security, but the article argues that this assumption inflates cost and slows investigations. In practice, the problem is not only data volume, but the mismatch between what is collected and what actually supports detection, especially where authentication, privilege, and workload identity signals are involved.

For IAM and NHI programmes, the governance question is whether high-value identity events are being preserved for alerting and investigation while lower-value operational noise is pushed to cheaper storage. That distinction matters in SIEM design, in cloud logging strategy, and in how teams balance compliance retention with analyst usability.

Key questions

Q: How should security teams decide which telemetry belongs in the SIEM?

A: Teams should keep only the telemetry that supports active detection, response, or short-horizon investigation in the SIEM. Everything else should move to lower-cost storage with clear retention and retrieval rules. The test is simple: if a log source cannot trigger, confirm, or materially speed up a detection workflow, it does not belong in the hot analytics tier.

Q: Why do IAM and NHI events need special treatment in log pipelines?

A: IAM and NHI events often provide the earliest evidence of abuse because they show who or what accessed a system, when privileges changed, and whether credentials were used unexpectedly. If those events are buried inside bulk telemetry, teams lose the clearest path from access to impact. Identity signals need priority because they are inherently high-value.

Q: What breaks when every log is stored in the same high-performance tier?

A: Searches slow down, storage costs rise, and analysts spend more time filtering noise than finding threats. The organisation also becomes less agile because adding a new source increases volume without proving value. A single analytics tier makes sense only when nearly every event class contributes directly to detections, which is rarely true in practice.

Q: How do security teams know whether agent telemetry is actually working?

A: Telemetry is working when it captures tool calls, lifecycle events, and message traces in a way that supports investigation and correlation. If logs omit tool execution or are easy to alter, the control has failed. Durable output with redaction and forwarding to monitoring systems is the practical test.

Technical breakdown

Why telemetry stacks become unmanageable

Security teams often expand collection because compliance, visibility, and future threat hunting all seem to justify keeping everything in the hottest storage tier. That creates a detection environment where search performance degrades as volume rises, even when the extra data adds little analytic value. The article’s core point is that the problem is architectural, not just operational: if every source is treated as equally important, the stack grows faster than the team’s ability to use it.

Practical implication: classify data by detection value before ingestion, not after the SIEM starts struggling.

Mapping authentication and privilege events to collection strategy

The most defensible real-time telemetry is the data that supports concrete scenarios such as authentication anomalies, privilege escalation, and unusual access paths. System performance metrics or other low-signal events may still be useful for retention or forensic review, but they do not belong in the same high-cost pipeline as the signals used to catch abuse. For identity teams, this means separating account activity, token use, and privileged operations from general infrastructure noise.

Practical implication: reserve real-time analytics for identity and privilege signals that can actually trigger detections.

Streaming detection changes the storage model

A modern telemetry strategy does not require every log to sit in the same expensive analytics tier. Streaming detection can evaluate relevant events as they arrive, while less time-sensitive records move to low-cost object storage for compliance and investigation. This shifts the design from hoarding logs to routing them by purpose, which improves search speed and reduces the chance that analysts drown in low-value data.

Practical implication: design tiered telemetry pipelines so detection data and retention data follow different paths.

NHI Mgmt Group analysis

Telemetry sprawl is a governance failure when collection is not tied to a detection outcome. Security teams often justify ingestion with compliance or visibility arguments, but those are not the same as operational need. When the stack expands faster than detection value, analysts pay the cost in search latency, storage expense, and missed signal. The right governance question is not how much can be collected, but which events actually justify real-time treatment. Practitioners should treat collection scope as a control decision, not a default.

Identity and privilege events deserve priority because they are the shortest path from access to impact. Authentication patterns, privilege escalation, and token use are the signals most likely to expose misuse before damage spreads. That makes IAM and NHI telemetry fundamentally different from bulk system metrics or generic infrastructure logs. If those identity events are buried in the same pipeline as everything else, the organisation is effectively weakening its own detection posture. Practitioners should separate high-value identity telemetry from the rest of the stack.

Stack reduction is now a detection quality issue, not just a cost optimisation exercise. The article’s argument aligns with a broader security reality: better detection depends on cleaner signal, not just more data. High-volume storage can hide weak analytic design, while tiered routing forces teams to define what each event class is for. In NHI-heavy environments, that distinction matters because service account and API activity can be lost in volume unless the pipeline is engineered around it. Practitioners should use telemetry design to sharpen, not merely expand, detection capability.

Detection scope leakage: this is what happens when teams keep every log in the same high-performance stack even though only a small subset supports real detections. The article shows that the failure is not the absence of telemetry, but the absence of a purpose-built collection model. Once that leakage begins, investigations get slower and more expensive, and important identity events lose prominence. Practitioners should define which sources belong in the analytic tier and which belong in retention only.

What this signals

Detection-scope discipline is becoming a programme-level requirement, not a tuning exercise. Teams that cannot separate identity and privilege telemetry from low-value infrastructure noise will keep paying for storage they cannot operationalise, while their most important events become harder to find.

In identity-heavy environments, the signal economy is becoming more important than raw volume. The practical move is to reserve the fast path for authentication, access, and secret-use events, then align longer-retention logging to compliance or forensic needs rather than real-time analytics.

As telemetry estates grow, the security value of each data source should be reviewed the same way access is reviewed: by purpose, urgency, and business need. That is especially true where secrets, service accounts, and workload identities are concerned, because those events often carry the earliest indicators of misuse.

For practitioners

• Separate detection data from retention data Classify each log source by whether it supports an active detection scenario, then route non-detection data to low-cost storage instead of the SIEM.
• Prioritise identity and privilege telemetry Keep authentication, privilege escalation, token usage, and account activity in the fast analytics path because those events are the most likely to reveal abuse.
• Define ingestion rules by use case Require every new data source to justify the detection or investigation use case it serves before it is added to the real-time stack.
• Measure search latency and analyst workload Track how long queries take across the existing telemetry estate and use that metric to identify sources that add cost without improving triage.

Key takeaways

• Telemetry overload is not just expensive, it hides the events analysts need most to detect identity and privilege abuse.
• The most useful security data belongs in the fast analytic path only when it can drive a real detection or investigation outcome.
• Tiered collection, not indiscriminate ingestion, is the practical way to shrink the stack without weakening visibility.

Key terms

• Telemetry sprawl: Telemetry sprawl is the accumulation of too many logs, metrics, and events across too many systems for security teams to use effectively. It usually happens when collection decisions are driven by fear of missing something rather than by a clear detection or investigation purpose.
• Detection tier: A detection tier is the high-performance part of a security data pipeline where events are searched, correlated, and turned into alerts. It should contain only the data that materially supports near-real-time detection or investigation, not every available record.
• Retention tier: A retention tier is lower-cost storage used to preserve data for compliance, audit, and later forensic review. It keeps information available without forcing it into the expensive, analyst-facing part of the pipeline.
• Privilege telemetry: Privilege telemetry is the set of logs and events that show when access is granted, changed, or used in a high-risk way. It includes authentication activity, role changes, token use, and other signals that can reveal abuse before it spreads.

What's in the full article

Abstract Security's full article covers the operational detail this post intentionally leaves for the source:

• How the vendor structures security data pipelines to keep detection data separate from cheaper retention storage
• Practical examples of which log sources are worth keeping in the high-performance analytics tier and which are not
• The logic behind reducing SIEM volume without sacrificing visibility for authentication and privilege monitoring
• The vendor's explanation of the "log lemming effect" and how it changes collection decisions

👉 Abstract Security's full post explains how to reduce telemetry volume without losing detection value

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and identity lifecycle fundamentals. It helps practitioners connect identity controls to operational security decisions across modern environments.