Security Log Schemas Compared: Choosing a Normalisation Target for Detection

Every security data platform has to answer one design question early: which schema the events land in, because that choice decides how detections are written, how portable they are and how much a change of tool will cost later. Four open formats dominate the answer: ArcSight Common Event Format (CEF) and its QRadar cousin LEEF, the Elastic Common Schema (ECS), the Open Cybersecurity Schema Framework (OCSF) and the OpenTelemetry semantic conventions with the OpenTelemetry log data model. They were built for different jobs, at different times, and they are not interchangeable. This guide compares them on structure, identity coverage, versioning, transport and failure modes, then gives decision criteria and a worked migration scenario. It is for SOC architects, detection engineers and data pipeline owners choosing a normalisation target or living with several at once. Schemas are listed alphabetically; order is never a ranking. Versions and status were checked on the official sources on 14 November 2026.

Key takeaways

• CEF and LEEF are wire formats, not data models. They define a header and a flat key=value dictionary carried over syslog; they say little about how a logon differs from a process start, so meaning lives in each vendor's mapping.
• ECS and OCSF are event models. ECS categorises with event.kind, event.category, event.type and event.outcome on a nested field tree; OCSF assigns every event to one numbered class with typed objects and enumerations.
• OpenTelemetry is the observability side converging with security. Elastic and OpenTelemetry agreed in April 2023 to merge ECS into the OpenTelemetry semantic conventions, and ECS fields have since flowed into the HTTP, network, process, file and url namespaces.
• Identity is where the schemas differ most. OCSF has an IAM category and user.type_id values for service and system accounts; ECS has the iam and authentication categories; CEF has suser and duser strings; OpenTelemetry user.* attributes are still marked development.
• Pick one canonical schema for detection, translate at the edges. Most estates keep CEF or LEEF for legacy appliances and OpenTelemetry for applications, and normalise both into ECS or OCSF where rules run.

The four schemas at a glance

The table summarises what each specification actually is. Much of the confusion in schema debates comes from comparing a transport format with an event model.

CEF and LEEF: syslog-era wire formats

Architecture in words. A CEF event is a single line: CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|Extension. The header names the source and the event class ID the vendor chose; the extension is a flat set of keys from the CEF dictionary such as rt (receipt time), src and dst, spt and dpt, suser and duser, act, outcome and cat. Anything without a dictionary key goes into six custom strings, cs1 to cs6, each with a cs1Label style label. The implementation standard sets limits (63 characters for vendor and product, 1,023 for the event class ID, 512 for Name, 4,000 for each custom string), requires UTF-8 throughout, and defines escaping: \| in the header, \= in the extension, \\ anywhere, and \n or \r for newlines in extension values only. Severity is 0 Unknown, 1 to 3 Low, 4 to 6 Medium, 7 to 8 High and 9 to 10 Very-High. CEF:0 means specification 0.1; CEF:1 means 1.x, which added IPv6 in address fields and Long values for byte counters.

LEEF follows the same idea for IBM QRadar: LEEF:1.0|Vendor|Product|Version|EventID| then tab-separated attributes such as src, dst, srcPort, usrName, cat and sev. LEEF 2.0 adds a delimiter field so a source can use ^ or a hex-coded character instead of tab. IBM's documentation states that attribute order is not enforced, the event ID is limited to 255 characters and events must be UTF-8.

Threat model. These formats assume a network appliance or security product pushing events to a collector over syslog. They protect nothing themselves: integrity and confidentiality depend on the transport (syslog over TLS or a connector), and a spoofed UDP sender produces a perfectly valid CEF line.

Failure modes. The meaning of an event lives in the vendor's Device Event Class ID, so "failed logon" is a different string per product. Custom strings drift: cs3 is a policy name in one product and a URL category in another, and only the label says which. Escaping errors truncate events silently at the first unescaped pipe or equals sign. Severity is the vendor's own judgement mapped to a 0 to 10 scale.

Controls. Keep a per-product parsing table that maps Device Event Class IDs and csN labels to your canonical fields; test parsers against escaped samples; send over TLS; alert on parse failure rates. The Network Device Syslog and Configuration Change Detection Guide covers the device side.

Elastic Common Schema: categorised field tree

Architecture in words. ECS defines field sets (for example event, user, source, destination, process, file, host, cloud) and, for every field, a name, an Elasticsearch data type, a description, an example and a level (core or extended). Four fields carry the categorisation: event.kind (alert, asset, enrichment, event, metric, state, pipeline_error, signal), event.category (values including api, authentication, configuration, iam, intrusion_detection, malware, network, process, session, threat, vulnerability and web), event.type (values including access, admin, allowed, change, creation, deletion, denied, start, end, user and group) and event.outcome (success, failure, unknown). A failed sign-in is therefore event.category: authentication, event.type: start and event.outcome: failure, regardless of product. ECS is deliberately permissive: fields with no ECS home may be added under custom names, and the documentation gives naming guidance for them. Version 9.4.0 introduced field maturity levels (alpha, beta, GA) and entity relationship fields; 9.5.0 added vulnerability.status and host.tags.

Threat model. ECS assumes events are already collected, typically by an agent or pipeline, and indexed for search. Its value is correlation across sources by shared field names, so the risk is semantic rather than transport: two pipelines that set user.name differently break joins.

Failure modes. Categorisation is multi-valued (event.category and event.type are arrays), so rules that test a single value miss events with two. Custom fields at the top level collide with future ECS fields. Pinning is often forgotten: ecs.version should record which release the document follows. Major ECS versions follow major Elastic Stack versions, so upgrades can arrive with a platform upgrade rather than a schema decision.

Controls. Validate documents against the published ECS field definitions in the pipeline, namespace custom fields under an organisation prefix, and test detections against multi-valued category arrays. The Detection as Code Guide covers schema tests in CI.

OCSF: class-based event model

Architecture in words. OCSF puts every event in exactly one event class inside one of eight categories: System Activity (1), Findings (2), Identity and Access Management (3), Network Activity (4), Discovery (5), Application Activity (6), Remediation (7) and Unmanned Systems (8). Each class defines its own activities and required attributes; reusable objects (user, actor, device, process, session, metadata) and enumerations with numeric IDs replace free text. Profiles add attribute sets across classes and extensions add vendor or platform content without changing the core. The OCSF Guide explains the Base Event, type_uid arithmetic and mapping decisions in depth; this page does not repeat them.

Threat model. OCSF was designed for multi-vendor security data lakes where producers and consumers are different organisations. It assumes mapping happens at a producer, a pipeline or a lake service, and that consumers rely on class and enumeration IDs rather than vendor strings.

Failure modes. Choosing the wrong class (a role attachment mapped to API Activity instead of Account Change) hides the event from identity rules. Fields dumped into unmapped look present but are not queryable in a stable way. Minor releases add attributes, so a lake with mixed metadata.version values needs version-aware queries.

Controls. Publish a class selection table for each source, reject events missing required attributes, record metadata.version, and track the share of events with populated unmapped.

OpenTelemetry: observability conventions growing into security

Architecture in words. OpenTelemetry has two parts that matter here. The log data model (status Stable) defines the envelope of a log record: Timestamp and ObservedTimestamp in nanoseconds, TraceId and SpanId for correlation with traces, SeverityText, SeverityNumber (1 to 4 TRACE, 5 to 8 DEBUG, 9 to 12 INFO, 13 to 16 WARN, 17 to 20 ERROR, 21 to 24 FATAL), Body, Resource, InstrumentationScope, Attributes and EventName, where a record with a non-empty name is an Event. The semantic conventions (1.45.0) then name the attributes inside, by domain: HTTP, network, database, messaging, RPC, cloud providers, system, exceptions and more, with Generative AI conventions now kept in a separate repository. Telemetry schemas (status Stable) give each conventions release a schema URL and an immutable schema file listing transformations such as attribute renames, which the Collector's schema translation can apply so backends receive the version they expect.

The convergence with ECS is visible in the conventions changelog: version 1.21.0 (July 2023) made a breaking change to adopt ECS attributes in HTTP conventions and renamed net.* attributes to align with ECS, and later releases added ECS process, file and url fields. Security-specific namespaces exist (user.*, security_rule.*, source.*, destination.*, tls.*, dns.*), but every user.* and security_rule.* attribute is marked development in the registry as of November 2026.

Threat model. OpenTelemetry assumes instrumented applications and infrastructure exporting traces, metrics and logs over OTLP. Its strength for security is context: a log line carries the trace and resource that produced it. The OpenTelemetry GenAI Semantic Conventions for Security Teams shows the same pattern for AI agents.

Failure modes. Development-status attributes can be renamed between releases. Without a security event taxonomy, an authentication failure is whatever EventName the developer chose. Bodies may carry secrets or personal data unless the Collector redacts them.

Controls. Pin the schema URL, run schema translation in the Collector, define an internal EventName catalogue for security events, and add redaction processors before export.

Side-by-side comparison

Where identity fits

Non-human identities make schema choice concrete. A service account signing in to a cloud API, an API key used from a new network and a workload token exchanged for another must all be recognisable as machine activity to be detected. In OCSF, user.type_id distinguishes User (1), Admin (2), System (3) and Service (4), so one rule can target service accounts across sources. In ECS you filter on user.* plus a categorisation of iam or authentication, and machine status usually needs enrichment. In CEF the account is a string in suser or duser, so machine classification depends on a lookup table. OpenTelemetry records the calling service in Resource, which is useful context but not an identity claim. Whatever the target, enrich events with the owner, type and credential ID from your NHI inventory before rules run. See the Cloud Logging and Audit Trail Guide for the audit sources that carry these identities.

Decision guidance: which schema for which job

Trade-offs. OCSF gives the strictest contract and the most identity structure, at the cost of mapping effort and class selection judgement. ECS is quicker to adopt where content already exists and tolerates partial mapping, which is both its convenience and its risk. CEF and LEEF need the least from producers and the most from parsers. OpenTelemetry offers context no other format carries natively but leaves security event typing to you.

Worked scenario: a failed service account sign-in across four formats

A detection engineer wants one rule: "repeated failed sign-ins by a service account from a new network". Four sources report the same failure.

1. Firewall VPN (CEF): header event class ID is vendor specific; the account is in suser, the address in src, and the result in outcome as a string. The pipeline maps the class ID to "authentication failure" using a per-product table and looks the account up in the NHI inventory.
2. Directory (ECS): the agent already sets event.category: authentication, event.outcome: failure, user.name and source.ip. Enrichment adds a machine-account flag.
3. Cloud audit log (OCSF via the lake service): class Authentication (3002), status_id failure, user.type_id 4 Service, src_endpoint.ip. No enrichment is needed for type.
4. Internal API gateway (OpenTelemetry): a log record with EventName from the internal catalogue, user.id, client.address and the gateway service in Resource. The Collector maps it to the canonical schema.
5. The rule: written once against the canonical schema, with a test fixture from each of the four sources in CI. When the cloud provider adds a new field in a later OCSF minor version, the test suite catches any change in behaviour.

Implementing a canonical schema

1. Inventory sources and formats. Record for each source its native format, version signal, volume and owner. The SIEM Log Source Onboarding Checklist gives a template.
2. Choose the canonical schema and pin a version. Write the decision down with the version (for example OCSF 1.9.0 or ECS 9.5.0) and an upgrade policy.
3. Write mapping specifications. One table per source: native field, canonical field, transformation, and what goes to raw or unmapped.
4. Pilot with identity sources first. Directory, IdP and cloud audit logs give the fastest detection return. Run old and new pipelines in parallel and compare rule hits.
5. Add schema validation in the pipeline. Reject or quarantine events missing required fields; count, never drop silently. The Security Data Pipeline Guide covers where to run it.
6. Port detections and add fixtures. Convert rules to the canonical names (Sigma field mappings help; see the Sigma Rules Guide) and keep a sample event per source as a test.
7. Roll back safely. Keep raw events for the retention period so any source can be re-mapped, and keep the old index until rule parity is proven.

Detection and measurement

• Parse success rate: share of events per source that parse and validate; target above 99 per cent for identity sources.
• Required field coverage: percentage of events with user, source address, outcome and time populated.
• Unmapped share: proportion of events with fields in unmapped or custom strings; falling over time shows mapping maturity.
• Version spread: count of distinct schema versions in the store; more than two signals upgrade debt.
• Rule portability: share of detections that reference only canonical fields.
• Ingest latency: difference between event time and observed or processed time, per source.

Common failure patterns

• Comparing a transport with a model. Choosing "CEF versus OCSF" as if both were targets leads to rules written against vendor strings.
• Normalising without pinning a version. Mixed ECS or OCSF versions in one store make the same query return different results by date.
• Treating custom slots as fields. Rules that read cs2 or unmapped attributes break when a vendor changes its connector.
• Dropping raw data. Without the original event, a mapping error cannot be corrected after an incident.
• Ignoring machine identity. Normalised schemas that never classify service accounts leave machine credential misuse in the noise; the Microsoft Midnight Blizzard breach involved a legacy test OAuth application granting itself mailbox access; the audit logs held the activity, but app consent and role assignment events only stand out when machine identities are typed and alerted on.
• Logging secrets. Application bodies and custom strings sometimes carry tokens; redact in the pipeline.

Framework mapping

Practitioner checklist

• Separate wire formats (CEF, LEEF, OTLP) from the canonical event model in your architecture diagram.
• Choose one canonical schema for detection and record the pinned version and upgrade policy.
• Maintain a mapping specification per source, including Device Event Class IDs and custom string labels.
• Validate events against the schema in the pipeline and quarantine failures.
• Classify human and machine identities in every authentication and IAM event.
• Keep raw events long enough to re-map after a parser fix.
• Keep one test fixture per source per detection and run them in CI.
• Redact secrets and personal data before storage, especially from OpenTelemetry bodies.
• Report parse success, field coverage, unmapped share and version spread monthly.

Standards and references

• OCSF: ocsf-schema repository
• Elastic: Elastic Common Schema (ECS) reference
• OpenText ArcSight: Implementing ArcSight Common Event Format (CEF)
• IBM: LEEF event components
• OpenTelemetry: Logs data model
• OpenTelemetry: Semantic conventions
• OpenTelemetry: Telemetry schemas
• OpenTelemetry: ECS and OpenTelemetry semantic conventions convergence announcement

Related NHI Mgmt Group resources: OCSF Guide · Security Data Pipeline Guide · Detection Engineering Guide · CIS Control 8 Audit Log Management Guide