Memory-only queues break down when syslog-ng or the host crashes, because queued messages are not durable. They also struggle when destinations are down or overloaded for long enough that the queue fills, which can lead to flow-control and backpressure. In practice, that means observability data can be delayed, rejected, or lost unless the pipeline has disk-backed buffering and enough capacity for peak conditions.
Why This Matters for Security Teams
Memory-only buffering is a reliability choice with security consequences. When syslog queues live only in RAM, the logging pipeline inherits the weakest properties of volatile memory: any restart, crash, or forced reboot can erase events that had not yet been forwarded. That creates blind spots exactly when teams most need continuity, such as during outage recovery, destination failure, or an active investigation.
The bigger operational issue is that syslog buffering is not just about smoothing traffic, it is also part of how the organisation preserves telemetry under stress. If the downstream collector is slow or unavailable long enough, a memory queue can saturate and start rejecting or delaying messages. At that point, the logging path becomes part of the incident rather than a record of it. For teams that depend on syslog for audit trails, security monitoring, or forensics, that is a material loss of evidence. In practice, many teams discover the gap only after a restart or long outage has already erased the very messages they expected to review later.
How It Works in Practice
A memory queue absorbs bursts by holding messages in RAM until the destination can accept them. That works well for short spikes, but it has two hard limits. First, RAM is not durable, so anything still queued when the process or host fails is gone. Second, the queue can only buffer up to the available memory budget. Once the buffer fills, the sender may be slowed by flow control or messages may be dropped, depending on the configuration and the rest of the pipeline.
In practical terms, the failure mode depends on what sits downstream of syslog:
- If a remote log receiver is briefly slow, memory buffering may smooth the spike with no visible impact.
- If the receiver is unavailable for longer than the queue can hold, the pipeline begins to accumulate backpressure.
- If the host crashes, queued events disappear even if the destination comes back cleanly.
- If logging volume spikes during an incident, the queue can fill faster than operators expect, especially on busy hosts.
Disk-backed buffering changes the outcome because it survives process restarts and host failures, so delayed messages can still be delivered once the destination recovers. That does not remove all risk, because storage capacity, filesystem health, and write latency still matter, but it materially reduces the chance that transient instability turns into irreversible log loss. The choice is therefore less about “performance versus durability” in the abstract and more about whether the pipeline can tolerate losing the evidence trail during the exact periods when it is under the most stress. These controls tend to break down on high-volume collectors with undersized disks or aggressive retention settings because the buffer fills before the destination recovers.
Common Variations and Edge Cases
Tighter buffering usually improves responsiveness, but it also increases the chance that queued telemetry disappears during failure, so organisations have to balance low-latency handling against evidence retention.
A few edge cases matter in real deployments:
- Short-lived test environments may accept memory-only buffering if losing logs is operationally tolerable.
- Production systems with audit, incident response, or compliance requirements usually need durable buffering for at least the critical log paths.
- Disk buffering helps, but it still fails if local storage is full, degraded, or excluded from monitoring.
- Very large spikes can exhaust both memory and disk queues, so sizing must reflect peak failure conditions rather than average traffic.
There is no universal standard for the “right” queue type, because the answer depends on how critical the logs are, how long downstream outages can last, and how much loss the business can absorb. For low-value telemetry, memory may be acceptable. For security logging, audit trails, and investigation data, durable buffering is the safer default because it preserves the chain of evidence across crashes and prolonged destination outages. The main trade-off is that durability introduces storage management and monitoring overhead, which teams often underestimate until the buffer silently approaches exhaustion.
Risk and Threat Considerations
The material risk is loss of observability at the point where trust in the logging pipeline matters most. Memory-only queues create a single point of failure for evidence preservation, and prolonged downstream outages can turn a temporary delivery problem into permanent log loss.
Failure mechanism: The queue is emptied by any process or host restart, and it fills when the downstream target stays unavailable or slow. Once flow control engages, logs can be delayed, rejected, or dropped, which weakens incident detection and post-incident reconstruction.
Impact: Security teams may lose audit continuity, miss indicators of compromise, or be unable to prove what happened during an outage window. That can impair investigations, compliance evidence, and recovery decisions.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST CSF 2.0 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.PT-5 — Resilient Infrastructure | Syslog buffering supports resilient telemetry delivery under failure. |
| DE.CM-8 — Detection Processes | Loss of queued syslog events weakens security monitoring visibility. | |
| Recommendation — Use resilient logging paths that preserve telemetry during outages and restarts. Ensure logging pipelines retain events needed for continuous detection coverage. | ||
| CIS Controls v8 | 8.2 — Audit Log Management | Syslog queues directly affect retention and delivery of audit logs. |
| 8.3 — Ensure Adequate Logging and Monitoring | Buffered syslog is part of maintaining usable monitoring data. | |
| Recommendation — Configure audit logging to survive collector outages and system restarts. Size and monitor log buffers so security telemetry is not dropped under load. | ||
Practitioner Guidance
What to prioritise: Treat any syslog path that carries audit, authentication, or incident-response data as durability-critical. If loss of those events would weaken an investigation, memory-only buffering is the wrong default.
What to verify: Confirm how the queue behaves on restart, what happens when the destination is unavailable for hours rather than seconds, and whether the configured capacity matches worst-case logging volume. Also verify that disk-backed buffering is on a monitored filesystem with enough headroom to survive the expected outage window.
Decision rule: If the log stream supports security monitoring or compliance evidence, prefer disk-backed buffering and alert on queue growth. If the stream is purely operational and loss is acceptable, memory may be adequate, but only with an explicit documented exception.
Practitioner takeaway: The important judgement is not whether buffering is fast enough, it is whether the pipeline can keep its evidence intact when the collector, destination, or host is failing.
