How should security teams handle filtering and rewriting when log data contains deeply nested JSON or OpenTelemetry fields?

Security teams should move from separate filter, parser, and rewrite steps to a structured approach that can inspect and modify typed fields directly. That reduces configuration sprawl, makes rules easier to read, and avoids unnecessary parsing overhead. It also helps when the goal is to curate log data, remove noise, and route only the fields needed for analysis or storage efficiency.

Why Structured Field Handling Matters for Nested Logs

Deeply nested JSON and OpenTelemetry payloads change the problem from simple text matching to data handling. When teams keep filtering, parsing, and rewriting as separate layers, they often create brittle rules that depend on field order, parser success, or duplicated logic across pipelines. A structured approach is more reliable because it operates on typed fields directly, so teams can curate what they ingest, reduce log noise, and preserve the parts of the record needed for investigation, retention, or cost control.

That matters operationally because the wrong field-handling model can either drop useful telemetry or keep too much irrelevant data. In log pipelines, the risk is rarely that the event exists, but that teams cannot consistently shape it before it becomes expensive to store or difficult to search. The best designs make filtering and rewriting predictable at the record level, not at the text-string level. In practice, many security teams only discover that weakness after nested schemas have already fragmented their routing rules and made pipeline changes harder to validate.

How Structured Filtering and Rewriting Works in Practice

The key shift is to treat the log event as a structured object from the start. Instead of extracting a string, applying a parser, then rewriting the result, the pipeline should inspect the nested fields directly and make decisions on the original typed representation. That approach is especially useful when logs contain arrays, embedded attributes, or OpenTelemetry resource and span fields that need selective retention rather than blanket processing.

For security teams, the practical benefit is not just cleaner configuration. It is also fewer failure points. A rule that reads a field from the structured event is less dependent on delimiter assumptions, escaping behaviour, or the exact shape of the serialized payload. It is also easier to reason about because the intent is visible in one place: what to keep, what to drop, and what to rewrite.

Use field-aware conditions when the source already exposes JSON or OpenTelemetry attributes.
Apply rewrite logic to the structured record before serialization, so the same rule works across downstream outputs.
Keep parsing only for sources that truly arrive as opaque text, not for records that are already typed.
Validate that nested keys, arrays, and absent fields behave consistently, especially when schemas evolve.

This model is strongest when the goal is curation rather than enrichment. It breaks down when the upstream source is inconsistent, when field names are unstable across teams, or when a platform cannot preserve typed structure through the full route to storage and analysis.

Where Nested Schemas Change the Filtering Tradeoff

Tighter field-level handling often improves precision, but it also increases dependence on schema stability, so organisations must balance cleaner pipelines against the operational cost of change management. That tradeoff becomes visible when different applications emit similar events with different nesting depth, optional keys, or OpenTelemetry conventions.

One common edge case is when teams assume a field exists at a fixed path. With nested JSON, a missing object or renamed attribute can cause a rule to stop matching silently, which is worse than a noisy failure because the pipeline still appears to work. Another edge case is selective redaction: rewriting a nested field is safer than rewriting the whole blob, but only if the team can prove the transform does not remove context needed for detection or incident review. OWASP’s Non-Human Identity Top 10 is relevant where those log fields include service identities, tokens, or automation context, because the logging design then affects both observability and identity governance.

Guidance here is partly consensus and partly implementation-specific. The consensus view is that structured processing is preferable when the source already provides structure. The less settled question is how far teams should push rewriting into the log pipeline versus leaving some transformation to downstream search or SIEM layers.

Risk and Threat Considerations

Nested log handling creates exposure when teams rely on brittle parsing or rewrite logic to decide what gets retained, masked, or routed. If those decisions are applied inconsistently across branches of a pipeline, sensitive fields can leak into storage, or important telemetry can be stripped before detection rules see it.

Failure mechanism: The weakness usually appears when a pipeline treats structured data as text, or when it assumes a fixed field path that no longer matches the emitted schema. Attackers and abusive insiders can benefit from that gap if security-relevant fields are omitted from monitoring, if redaction misses nested attributes, or if malformed records bypass transformation logic and preserve sensitive content.

Impact: The result can be incomplete investigations, incorrect retention, unnecessary storage growth, or exposure of credentials, identifiers, and operational context that should have been removed before indexing.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATT&CK and OWASP Non-Human Identity Top 10 address the attack and risk surface, while CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
CIS Controls v8	8 — Audit Log Management	Nested log shaping supports selective retention, masking, and routing of audit data.
	13 — Network Monitoring and Defense	Field-aware routing and filtering support usable monitoring data for detection workflows.
Recommendation — Standardise log field handling so audit data stays searchable without exposing unnecessary detail. Tune monitoring pipelines so nested fields reach detection tools in a usable form.
NIST CSF 2.0	DE.CM-8 — Vulnerabilities in External Dependencies are Monitored	Structured logs improve monitoring coverage and reduce blind spots in telemetry pipelines.
Recommendation — Use DE.CM-8 to keep telemetry visibility intact as schemas and log sources change.
MITRE ATT&CK	T1070 — Indicator Removal on Host	Log rewriting and selective retention can affect evidence preservation and analyst visibility.
Recommendation — Map log suppression or redaction patterns to T1070 where attackers target evidence removal.
OWASP Non-Human Identity Top 10	NHI-01 — Secrets and Credential Exposure	Nested logs may contain tokens, API keys, or service credentials needing field-level handling.
Recommendation — Scan structured logs for embedded secrets and redact them before indexing or export.

Practitioner Guidance

What to prioritise: Start with the fields that actually drive search, alerting, retention, and redaction decisions. If the pipeline cannot name those fields explicitly, it is probably still too text-centric for nested data.

What to verify: Confirm that missing nested keys, arrays, and schema version changes fail in a visible way. A rewrite rule that quietly stops matching is an observability defect, not a minor configuration issue.

What practitioners underestimate: The hardest part is often not the filter itself but the consistency of field paths across producers. When teams do not control schema discipline, the safest design is the one that assumes change and makes that change testable before rollout.

Practitioner takeaway: Treat nested log handling as schema governance, not just pipeline tuning, because precision only holds when the event structure is stable enough to support it.