Schema-less ingestion

An approach that accepts raw telemetry without forcing it into one fixed internal structure at the point of entry. This preserves source fidelity, which matters when identity, cloud, and workload logs need to be reinterpreted later for investigation or detection.

Expanded Definition

Schema-less ingestion is a telemetry intake pattern that accepts events, logs, and traces without forcing them into one fixed internal structure at the moment of entry. In NHI operations, that matters because service account activity, API gateway events, cloud audit logs, and agent tool calls rarely share the same fields, field order, or nesting depth.

The goal is not to eliminate structure. It is to delay rigid normalization until the data can be enriched, correlated, and validated against the investigation or detection use case. That approach aligns well with the NIST Cybersecurity Framework 2.0 emphasis on useful, reliable telemetry for detection and response, while preserving source fidelity for later parsing. In practice, schema-less pipelines often retain the raw payload alongside extracted fields so teams can reprocess historical data as threats, log formats, and agent behavior evolve.

Definitions vary across vendors on whether a schema-less system still applies lightweight validation, but no single standard governs this yet. The most common misapplication is treating schema-less ingestion as “no schema at all,” which occurs when teams fail to preserve raw records and then cannot reinterpret events after the logging format changes.

Examples and Use Cases

Implementing schema-less ingestion rigorously often introduces storage and processing overhead, requiring organisations to weigh forensic fidelity against parsing cost and index complexity.

• Preserving raw service account authentication events so investigators can later correlate them with suspicious token use and privilege escalation attempts.
• Ingesting cloud control-plane logs from multiple providers before normalising fields into a common detection model for NHI monitoring.
• Capturing agent tool execution telemetry, including prompts, actions, and API responses, before deciding which fields are relevant for NHI lifecycle review and threat hunting.
• Retaining raw webhook and application audit events so schema changes do not break historical investigations or retroactive detections.
• Supporting reprocessing workflows after a detection rule is updated, allowing teams to reparse older records without losing the original payload.

These patterns are particularly useful when working across identities, cloud workloads, and automated systems where one vendor’s log shape is not compatible with another’s. For implementation context, the Ultimate Guide to NHIs highlights how poor visibility and inconsistent credential handling leave many organisations unable to reconstruct events cleanly, which makes raw retention valuable during later analysis.

Why It Matters in NHI Security

Schema-less ingestion is important because NHI abuse often hides in telemetry that appears incomplete, inconsistent, or differently formatted across systems. If ingestion strips context too early, defenders may lose evidence of secret misuse, lateral movement, or agent-driven actions that only become obvious when multiple raw records are correlated.

This is especially relevant given NHI Mgmt Group data showing that only 5.7% of organisations have full visibility into their service accounts, and 96% store secrets outside of secrets managers in vulnerable locations including code, config files, and CI/CD tools, as reported in the Ultimate Guide to NHIs. Schema-less intake helps preserve the evidence trail needed to trace where a secret came from, how it was used, and which workload or agent invoked it. That said, the approach still needs governance, because raw retention without access controls can increase exposure if logs contain secrets or sensitive identifiers. In other words, the same flexibility that helps investigations can also widen the blast radius if ingestion, storage, and retrieval are not tightly controlled.

Organisations typically encounter the operational necessity of schema-less ingestion only after an incident review fails because the original logs were flattened, discarded, or parsed too narrowly, at which point the term becomes operationally unavoidable to address.

Related resources from NHI Mgmt Group

• Why is CVE-centric security becoming less reliable?
• Why do non-human identities make access reviews less effective?
• When does ephemeral authorization create less risk than persistent access?
• Should organisations use connector-less deployment for on-prem DSPM where possible?