Log curation is the process of reducing noise, extracting only relevant fields, and routing telemetry into the right destination for analysis or storage. It helps security and observability teams control ingest cost, improve search quality, and avoid carrying unnecessary data through the pipeline.
Expanded Definition
Log curation is the deliberate shaping of telemetry before it reaches analysis or long-term storage. It covers filtering, field selection, normalisation, enrichment, sampling, and routing, while excluding the broader work of detection engineering, SIEM tuning, or retention policy design.
The boundary that matters in practice is that curation changes what is collected and where it goes, but it should not silently erase evidence that another team depends on. Poor curation is often mistaken for “just trimming noise”, when in reality it can alter investigative context, compliance scope, and the fidelity of downstream analytics.
In security operations, the term is often used across cloud logs, application logs, identity events, and infrastructure telemetry. The challenge is to keep enough structure for search and correlation without paying to ingest every low-value field or duplicate record.
Examples and Use Cases
Common uses of log curation include:
- Dropping verbose debug fields from application logs before they enter a central platform.
- Routing high-value authentication and admin events into a security data store while sending routine operational events to cheaper archival storage.
- Normalising field names across services so searches and correlation rules work consistently.
- Extracting only security-relevant attributes from cloud audit trails to reduce ingest volume without losing auditability.
- Filtering duplicate or low-signal telemetry from agents and collectors so analysts see fewer redundant records.
The implementation trade-off is usually cost versus investigative depth. Heavier curation lowers storage and search burden, but it also increases the need to define which fields must be preserved for incident response, compliance review, and later forensics.
Security Implications
When log curation is too aggressive, the result is not just leaner telemetry. It can remove the fields that reveal actor identity, request path, source location, privilege context, or sequence of actions, which weakens correlation and delays triage.
Over-curation also creates blind spots that are hard to notice until an investigation is underway. Teams may still see that “something happened” without having the detail needed to determine whether it was a benign failure, a misconfiguration, or malicious activity.
Because curated pipelines often feed multiple consumers, one bad routing decision can have wider consequences than a single broken dashboard. A field dropped for cost reasons may also break detections, retention evidence, and incident reconstruction.
A practical observation is that the most common failure is not complete log loss but selective loss of context. Security teams usually notice this when searches become inconclusive even though volume looks healthy.
Domain and Governance Relevance
Log curation sits at the intersection of observability governance, security monitoring, and data minimisation. It is not just an engineering optimisation; it defines which telemetry becomes operational evidence and which signals are reduced to summaries.
In identity-heavy environments, curation has extra importance because authentication events, session metadata, token activity, and administrative actions often provide the only trace of privilege use. If those records are over-filtered, identity investigations become harder and anomalous access becomes easier to miss.
For non-human identities, curated telemetry often determines whether service account behaviour, API usage, and automated access patterns remain auditable end to end. That makes field selection and routing part of machine-identity visibility, not merely storage management.
When log curation is treated as a shared control rather than a local engineering choice, teams are more likely to preserve the evidence needed for detection, accountability, and recovery.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 address the attack and risk surface, while CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| CIS Controls v8 | 8 — Audit Log Management | Log curation directly shapes which audit data is retained and searchable. |
| Recommendation — Define log selection and retention rules that preserve security-relevant audit evidence. | ||
| NIST CSF 2.0 | DE.CM-1 — Monitoring for Unauthorized Activities | Curated telemetry feeds monitoring that depends on useful, timely event data. |
| PR.PT-1 — Audit/Log Records | Curation affects how audit records are generated, protected, and made available. | |
| Recommendation — Preserve the event detail needed to detect unauthorized activity reliably. Protect the log record chain so curation does not undermine evidence integrity. | ||
| OWASP Non-Human Identity Top 10 | NHI-05 — Observability and Detection | Telemetry curation affects visibility into service accounts, tokens, and automated access. |
| NHI-01 — Lifecycle and Ownership | Curation decisions should reflect ownership for non-human identity telemetry and evidence. | |
| Recommendation — Keep machine-identity telemetry sufficient for anomaly detection and investigation. Assign clear owners for which NHI-related logs are kept, routed, and reviewed. | ||
Related resources from NHI Mgmt Group
- How should security teams handle AI agents that need to log into SaaS applications?
- What breaks when hospitals do not log access to electronic patient data?
- How should security teams log privileged SSH access from bastion hosts?
- How should security teams log PostgreSQL activity without hurting performance?
