Telemetry intent gap is breaking security outcomes in unified pipelines

At a glance

What this is: This analysis argues that observability and security telemetry have different intents, so unifying them under one optimisation model can damage detection, investigations, and long-horizon evidence retention.

Why it matters: It matters to IAM, NHI, and SOC practitioners because identity events, privilege changes, and credential abuse often look low-value in the moment but become decisive evidence later.

By the numbers:

• Organizations applying pre-SIEM filtering and enrichment have reduced SIEM-bound data volume by 50 to 70 percent, cutting licensing costs by more than half.

👉 Read DataBahn's analysis of the telemetry intent gap in unified pipelines

Context

Security telemetry and observability telemetry are not interchangeable, even when they move through the same collectors and transport layers. The core issue is intent: one supports performance management, the other supports adversarial reconstruction. For IAM and NHI programmes, that distinction matters because authentication events, privilege changes, and credential use can look routine until they are tied to a later incident.

The article’s central claim is that cost-optimised pipelines often apply observability logic to security data, then suppress or compress the very events investigators need. That creates a Telemetry Intent Gap. For teams responsible for identity governance, the risk is that access evidence becomes incomplete just when identity abuse, lateral movement, or compromised credentials need to be proven.

Key questions

Q: How should security teams separate observability and security telemetry in practice?

A: Use different policy objectives for each stream. Observability data can be sampled, compressed, and retained for shorter periods because it exists to explain service health. Security telemetry should preserve the metadata needed for later correlation, forensic reconstruction, and identity investigation. If one pipeline serves both purposes, route by evidence value first and cost second, not the other way around.

Q: Why do identity and access logs need longer retention than operational logs?

A: Because their value often emerges only after an incident, not at the moment they were created. Authentication events, privilege changes, and workload access records can look routine until investigators need to reconstruct compromise paths, insider activity, or delegated access abuse. Without durable retention, those events stop being evidence and become missing context.

Q: What do security teams get wrong about AI-based log filtering?

A: They assume low recent alert volume means low security value. That assumption fits observability, where recent patterns often predict usefulness, but it fails for adversarial data. Attackers deliberately operate below detection thresholds and across long time horizons, so AI models must be tested against attack chains, not just noise reduction metrics.

Q: How do you know if a telemetry pipeline is failing security governance?

A: Look for broken event chains, missing identity context, stalled investigations, and a growing need to replay data from cold storage. Those are signs that the pipeline is optimising for efficiency while eroding evidence quality. If analysts cannot reconstruct who did what, when, and under which privilege state, governance has already degraded.

Technical breakdown

Why observability telemetry and security telemetry behave differently

Observability telemetry is built to answer whether systems are healthy right now, so sampling, compression, and short retention are rational design choices. Security telemetry has a different purpose: it must preserve evidence that may only become meaningful after a long dwell time or a later investigation. An authentication event, privilege escalation, or lateral movement signal can be irrelevant in the moment but essential months later. That is why optimisation for statistical usefulness can directly undermine adversarial truth.

Practical implication: design retention and routing based on evidence value, not only ingestion cost.

How AI relevance scoring can distort security pipelines

AI relevance scoring in telemetry pipelines often borrows observability assumptions, such as treating low-frequency events or the absence of recent alerts as low value. That works when the goal is reducing noise in operational data. It fails when adversaries intentionally stay below thresholds, spread activity over time, or combine weak signals into a meaningful chain. In security, the right question is not what was recently investigated, but what could matter in future reconstruction. Pattern compression can therefore remove context before it is ever needed.

Practical implication: validate any upstream filtering model against attack-chain reconstruction, not alert frequency alone.

What intent-aware telemetry architecture actually changes

An intent-aware pipeline separates optimisation logic by data purpose while still allowing shared transport. Security telemetry needs early context preservation, long-term evidentiary assumptions, and retention decisions informed by adversarial models. Observability data can still be compressed aggressively, but security-relevant events should keep the metadata needed for correlation, case building, and retrospective review. This is an architectural governance issue, not only a tooling issue, because the pipeline decides what can later be proved.

Practical implication: define separate policy paths for operational exhaust and security evidence before centralising data flows.

Threat narrative

Attacker objective: The attacker benefits from reduced forensic visibility, making identity abuse and movement harder to prove or unwind.

1. Entry occurs when low-value telemetry is filtered or compressed upstream because it does not trigger recent investigations or alerts.
2. Credential access or escalation evidence is lost when authentication events, privilege changes, or rare identity signals are treated as noise.
3. Impact appears later as incomplete investigations, broken event chains, and weakened ability to reconstruct attacker movement or identity abuse.

NHI Mgmt Group analysis

Telemetry intent separation is now a governance requirement, not a pipeline preference. Security teams have spent years trying to reduce cost by converging observability and security pipelines, but the article shows that the shared-infrastructure model only works when optimisation logic is separated. If the same retention policy governs both operational and adversarial telemetry, one of them will lose fidelity. For identity programmes, that loss often appears first in missing authentication context and weak incident reconstruction. Practitioners should treat intent separation as a control boundary, not a tuning choice.

Telemetry relevance cannot be decided by recent alert history alone. The article correctly identifies the danger in using short-term usage patterns to score security data, because adversarial behaviour is often low-and-slow and only becomes meaningful in hindsight. That makes the idea of “relevance” asymmetric across domains. In security, the absence of an alert is not evidence of absence. Teams should assume that some identity and access events are retained for future correlation, not immediate usefulness.

Telemetry intent gap: the failure mode is optimisation collapse, where cost logic overrides evidentiary logic. That concept is useful because it names the real problem rather than blaming any one pipeline technology. The collapse happens when compression, sampling, and AI scoring are applied uniformly to data with different purposes. For IAM, PAM, and NHI governance, the implication is that identity evidence must be protected from generic efficiency rules. Practitioners should explicitly exempt security-relevant identity events from observability-style pruning.

Identity telemetry is part of control assurance, not just detection plumbing. Access reviews, insider-threat investigations, and forensic reconstruction all depend on durable identity evidence. When telemetry layers discard context, governance teams lose the ability to validate who accessed what, when, and under what privilege state. That turns identity assurance into a retrospective guess instead of an auditable record. The practical conclusion is simple: identity telemetry deserves its own retention and enrichment policy.

AI-driven filtering must be adversary-aware before it is cost-aware. A model trained to eliminate noise from operational telemetry can make poor choices when faced with rare but security-significant identity events. That is especially risky in NHI and agentic AI environments, where credentials, tokens, and delegated actions may only surface as suspicious when correlated across time. Practitioners should require threat-model validation for any AI-based routing decision before allowing it to suppress security evidence.

What this signals

Telemetry intent separation will increasingly show up inside SOC, IAM, and platform governance discussions because cost pressure alone cannot justify pruning evidence-bearing identity data. Teams that rely on unified pipelines should expect more scrutiny of where enrichment happens, what gets retained, and which events are exempt from observability-style sampling.

For identity programmes, the operational signal is straightforward: if access logs, token usage, and privilege transitions cannot survive long enough to support reconstruction, the control environment is too aggressive. That makes durable identity evidence a resilience requirement, not a logging preference.

For practitioners

• Separate evidence retention from operational compression Create distinct policy paths for observability data and security telemetry so that authentication events, privilege changes, and other evidence-bearing records are not pruned by the same rules as performance logs.
• Classify identity events as long-horizon evidence Tag IAM, PAM, NHI, and federated access logs as future-correlation data, then set retention and indexing rules that preserve investigative context even when the event looks low value today.
• Validate AI filtering against attack reconstruction Test relevance scoring models against multi-stage identity abuse scenarios, including compromised credentials, low-and-slow escalation, and lateral movement, before they are allowed to suppress data.
• Preserve enrichment before routing decisions Attach asset identity, user or workload identity, and threat context before data reaches the SIEM so that routing decisions are made on enriched signal rather than raw volume.

Key takeaways

• The article’s core warning is that observability optimisation can quietly erase security evidence before investigators ever need it.
• The practical consequence is incomplete identity reconstruction, especially for authentication events, privilege changes, and low-and-slow attack chains.
• Security teams should separate evidence retention from cost-led telemetry pruning and validate AI filtering against adversarial scenarios.

Key terms

• Telemetry Gap: A missing or incomplete stream of security events that prevents the SIEM from building a reliable picture of activity. In hybrid environments, telemetry gaps often appear where identity context is weakest, turning visibility problems into detection and governance problems.
• Security Telemetry: The logs, events, and configuration data that let defenders understand and prove what happened in a system. For SaaS governance, telemetry is what turns an application from a black box into something the security team can review, alert on, and investigate.
• Observability Telemetry: Observability telemetry is data such as metrics, traces, and debug logs used to understand system health and performance. It is typically designed for aggregation, sampling, and short- to medium-term analysis rather than long-term forensic use.

What's in the full article

DataBahn's full article covers the operational detail this post intentionally leaves for the source:

• Pipeline design guidance for separating observability retention from security evidence retention
• Examples of how AI relevance scoring can suppress low-frequency but security-significant telemetry
• Practical discussion of enrichment timing, pre-SIEM filtering, and cost-control trade-offs
• The operational distinctions between stream enrichment and enrichment at query time

👉 DataBahn's full article covers the operational distinctions between observability and security telemetry

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to the operational and governance decisions their programmes depend on.