Prioritise the fields analysts query most often, then preserve completeness in a spillover structure or nested payload. That approach keeps the active query path fast without throwing away low-frequency data needed for investigations. The right balance is not full flattening versus full nesting, but selecting the engine-friendly layout that keeps both performance and fidelity acceptable.
Why OCSF Pipelines Usually Need a Two-Layer Data Shape
OCSF pipelines work best when the most queried fields are easy to scan, filter, and aggregate, while less common attributes remain available without forcing every record into a fully flattened shape. That usually means a hot path for operational analytics and a completeness layer for edge cases, investigations, and future use. The design question is not whether to preserve detail, but where to place it so the engine can still work efficiently.
Schema completeness matters because security telemetry often becomes more valuable after the fact, when analysts need context that was not part of the original query pattern. A pipeline that only keeps popular fields may be fast, but it can also erase relationships, nested objects, and long-tail attributes that later explain an event. A pipeline that preserves everything in the query path can become expensive to scan, harder to index, and slower to evolve as event volume grows.
The practical compromise is to treat the schema as a usability layer, not a storage mirror. Flatten what drives repeated analysis, keep the rest in a spillover structure or nested payload, and make sure the original event can still be reconstructed or inspected when needed. In OCSF, that balance is what lets teams support both operational search and forensic depth without forcing one requirement to completely dominate the other.
What Performance Optimisation Changes in Practice
Query performance improves when the columns analysts filter on most often are easy for the engine to prune, partition, and index. That is especially true for high-volume detection and reporting workloads, where small schema choices can multiply across millions of events. The more frequently a field appears in predicates, joins, dashboards, or alert logic, the more it belongs in the fast path.
Completeness, however, should not be treated as a separate archival concern. If important context only exists in deeply nested or secondary fields, the pipeline still needs a clear way to retain and retrieve it. A useful OCSF design keeps the query surface narrow enough for performance, but not so narrow that the event becomes analytically brittle.
For teams building or tuning these pipelines, this means the schema design is really an indexing decision disguised as a modelling decision. The best layout is the one that reduces repeated parsing work for common searches while preserving the original payload shape for uncommon investigations. In practice, that is often a combination of selected scalar fields, structured sub-objects, and a raw or semi-structured fallback.
How to Preserve Fidelity Without Slowing the Engine
Schema completeness is most useful when it protects against loss of meaning, not when it duplicates every attribute into the same query layer. Preserve fields that carry identity, time, source, outcome, or relationship context, then keep the long tail in a nested object or spillover field that analysts can open when needed. That keeps the event faithful without making every query pay the cost of maximum normalisation.
One common mistake is to flatten everything just because it feels operationally tidy. That can create brittle pipelines, duplicated values, and more expensive schema maintenance when upstream producers change shape. The opposite mistake is to preserve everything in nested form and expect downstream analysts to navigate it efficiently; that tends to hurt discoverability, alerting simplicity, and storage selectivity.
A good design is explicit about which fields are promoted because they are used often, and which fields are retained because they may matter later. When that split is documented and stable, teams can tune performance deliberately instead of arguing over whether completeness or speed won by accident.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5 sets the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | CM-2 — Baseline Configuration | OCSF pipeline field selection is a schema baseline decision. |
| CM-6 — Configuration Settings | Tuning query-facing fields is a configuration choice that affects performance and fidelity. | |
| AU-2 — Event Logging | OCSF pipelines exist to retain and structure event records for analysis and investigation. | |
| Recommendation — Define a baseline event shape and control when fields may be promoted or preserved. Tune pipeline schema settings to optimise common queries without dropping needed context. Capture event fields needed for analysis while preserving enough detail for later review. | ||
| ISO/IEC 27001:2022 | A.8.13 — Information backup | Preserving spillover or raw payload data supports retention of complete records. |
| A.8.15 — Logging | The topic concerns structured security telemetry and how much detail logging should keep. | |
| Recommendation — Retain complete event payloads in a recoverable form alongside query-optimised fields. Structure logs so frequent queries stay fast without losing investigative context. | ||
Practitioner Guidance
What to prioritise: Promote fields that drive the most common hunts, detections, and filters into the fast path first, then verify that the nested or spillover layer still contains enough context for incident review and enrichment.
What to verify: Check whether the fields you flattened are actually the ones used by queries with the highest frequency and cost. If not, you are optimising the wrong part of the pipeline and may have increased maintenance without improving speed.
Common mistake: Do not use a single design rule, such as “everything flattened” or “everything nested”, for all event types. OCSF pipelines usually need different trade-offs for high-volume detections, investigative search, and long-retention analytics.
Practitioner takeaway: The right balance is usually selective flattening plus faithful retention, because performance suffers when the hot path is too deep, and investigations suffer when the cold path is too shallow.
Related resources from NHI Mgmt Group
- How should security teams evaluate a query language for modern security data pipelines?
- What do teams get wrong when adopting a security data schema like OCSF?
- How should security teams balance AI model performance with data minimization requirements?
- How should security teams govern AI agents that query sensitive data in Snowflake?