What breaks when organisations rely on unstructured logs for detection engineering?

Why This Matters for Security Teams

detection engineering depends on consistent event meaning. When logs arrive as free-form text, teams lose the ability to reliably map activity to fields, users, assets, actions, and outcomes. That weakens correlation, makes alert logic fragile, and turns routine content development into a parsing exercise. It also complicates governance because security leaders cannot easily tell whether a gap is caused by coverage, data quality, or an actual absence of malicious activity.

From a control perspective, the issue sits squarely in telemetry quality and operational resilience. The NIST Cybersecurity Framework 2.0 expects organisations to know what data they collect, why they collect it, and how it supports detection and response. Unstructured logs make that harder because the same activity can be represented differently across sources, environments, and application versions. In practice, many security teams discover this only after a major investigation is slowed by brittle parsing rules and incomplete event context, rather than through intentional telemetry design.

How It Works in Practice

Unstructured logs usually reach the SIEM as text blobs that require regex parsing, custom tokenisation, or vendor-specific transforms before they can support detections. That creates a dependency chain: the source must emit stable text, the parser must remain aligned to that text, and the detection must assume the fields are populated correctly. Any change in format, timestamp layout, language, delimiter, or message order can break downstream correlation without breaking the log pipeline itself.

Operationally, teams end up maintaining several layers of normalisation:

• source-side formatting rules to force consistent output
• ingest-time parsing to extract entities such as user, host, action, and status
• schema mapping so analytics can reuse the same field names across sources
• quality checks to detect empty, shifted, or misclassified fields

This matters because detections are only as good as the structure beneath them. MITRE ATT&CK is useful here because many common techniques, such as valid account use or defence evasion, depend on reliable identity and process context to distinguish malicious activity from normal operations. The OWASP guidance on logging and monitoring also reflects a core principle: logs must be actionable, not just abundant, if they are to support investigation and alerting. When logs remain unstructured, every new use case consumes engineering time that should have gone into analytic coverage, tuning, and threat hunting.

Best practice is evolving toward structured, machine-readable telemetry at the source, with explicit field definitions and stable schemas. That may mean JSON logging, common event models, or application instrumentation that emits context rather than prose. It also means defining minimum logging requirements for critical systems so detection engineers are not forced to reverse-engineer meaning from ad hoc strings. These controls tend to break down when legacy applications, third-party SaaS integrations, or high-volume cloud services emit inconsistent message formats because the organisation cannot standardise telemetry at the source.

Common Variations and Edge Cases

Tighter log normalisation often increases implementation overhead, requiring organisations to balance richer detections against ingestion cost, application change risk, and parser maintenance. That tradeoff is real, especially in hybrid estates where some platforms cannot produce structured output without code changes or licensing uplift.

There is no universal standard for every environment yet. Some teams can move to structured logging quickly, while others need a staged approach that preserves legacy feeds during migration. In regulated or high-assurance environments, the priority is often traceability and evidentiary value, so keeping raw logs alongside normalised records may be necessary. That reduces the risk of losing original context when parsers fail or schemas evolve.

Identity-heavy environments also need special care. If unstructured logs obscure usernames, service principals, tokens, or session identifiers, detections for credential misuse become weak even if the volume of data is high. For cloud-native environments, this often intersects with NHI governance because service accounts, API keys, and workload identities generate the activity that defenders most need to correlate. The practical answer is not “log more”, but “log in a form that can be trusted, queried, and reused across detections.”

For teams aligning to governance expectations, the NIST Cybersecurity Framework 2.0 is a useful reference point for treating telemetry as a managed security capability rather than an afterthought.

Related resources from NHI Mgmt Group

• What breaks when organisations rely mainly on detection instead of prevention for social engineering and impersonation attacks?
• What breaks when organisations rely on detection after an agent acts?
• What breaks when organisations rely on audit logs instead of runtime enforcement?
• What breaks when organisations rely only on detection for synthetic content?