How do you index telemetry data?

Why telemetry indexing is really a schema and context problem

Telemetry indexing works best when you treat the index as a searchable representation of events, not as a second copy of raw logs. The key decision is which fields must be first-class for correlation, filtering, retention, and hunt workflows, then how to preserve enough original context that an analyst can still reconstruct what happened without rehydrating the entire source stream.

Start with a stable core of fields that are common across sources: time, source, asset, user or identity, event type, outcome, and severity where available. That gives you predictable joins and time-based analysis, while leaving room for source-specific fields to sit alongside the normalized core. For a practical data-pipeline view, the patterns in the Security Data Pipeline Guide map directly to this problem because indexing is downstream of collection, normalization, enrichment, and routing.

A good index also separates identity-bearing keys from display text. For example, a username, service account, device ID, cloud principal, or session key should be indexed as a structured value that can be queried consistently, rather than buried in a message blob. That is what allows hunts to answer “what touched this asset?”, “which actor made this change?”, and “what events belong to the same chain?” without depending on brittle text matching.

What to normalise before ingestion

Normalization is the part that makes telemetry useful across sources. Timestamps should be converted into one reference format, identifiers should be canonicalized, and repeated concepts should use the same field name even when vendors label them differently. If one source says src_ip and another says client.address, the index should still expose a common network field where possible.

The same principle applies to assets and identities. Hostnames, instance IDs, container names, user IDs, API clients, and service principals all need a controlled representation if you want reliable correlation. Without that, detections fragment across source-specific variants, and analysts end up manually translating data every time they pivot.

Good telemetry indexing also preserves hierarchy. A single event can belong to a process tree, a host, a session, a tenant, and a campaign or incident cluster. Index those relationships explicitly when they are available so you can support both narrow investigations and broader trend analysis. When telemetry flows through collectors and parsers, a pipeline reference such as the Security Data Pipeline Guide is useful because it shows how normalization and enrichment support later search quality.

Schema stability matters, but not at the expense of adaptability. If you freeze the schema too tightly, you will break ingestion whenever a new tool emits a field you did not anticipate. If you make it too loose, detections become inconsistent and search quality degrades. The practical balance is a stable core schema with controlled extension fields and a documented mapping layer for source-specific differences.

How indexing supports detection, hunting, and retention

Indexing is not only about search speed. It also shapes whether telemetry can support detections that rely on relationships, time windows, rarity, or sequence. A well-designed index lets you sort by event time, group by asset or actor, and query across many sources without losing the thread of a campaign or suspicious sequence.

Retention design also matters. Hot, warm, and archive tiers only work if the indexed metadata is good enough to let you find the right slice later. If the index omits key context, long-term retention becomes expensive because analysts must pull too much raw data to answer simple questions. If the index includes too much low-value text, search cost grows without improving detection value.

The best indexing schemes also make enrichment explicit. Geolocation, asset ownership, business criticality, and threat context can all help, but only if they are stored as searchable fields rather than trapped in annotations. That way, hunters can ask whether the same event matters differently on a domain controller, a production workload, or a low-value test system.

Practitioner Guidance

What to prioritise: Define the minimum core field set before adding source-specific detail. If the same question must be answerable from multiple tools, the index should expose one consistent representation of time, actor, asset, and event outcome.

What to verify: Check that normalization preserves analyst context, not just machine readability. If a search result cannot be tied back to the originating source, related entity, and event sequence, the index is too thin for investigation use.

Common mistake: Teams often optimize for ingestion convenience and later discover that the index cannot support correlation across identities, assets, or time zones. That usually means the schema was built around one source instead of the investigation workflow.

Practitioner takeaway: Design the telemetry index around the questions analysts will actually ask, then keep the schema stable enough for repeatable hunts and flexible enough to absorb new telemetry without breaking correlation.

Related resources from NHI Mgmt Group

• How do you know if zero trust is actually working for data?
• How do you know whether AWS data controls are actually working?
• How do you know if a data governance framework is actually working?
• How do you know if Copilot is exposing too much sensitive data?