What are the signs that a syslog-ng or AxoSyslog pipeline is failing?

Common signs include unreachable outputs, rejected socket connections, rising memory or disk queue usage, and increased worker latency. Drops on the source side and output events marked as dropped are especially important because they indicate data is being lost rather than merely delayed. These symptoms usually point to overload, misconfiguration, or downstream unavailability.

What failure looks like in a syslog-ng or AxoSyslog pipeline

A failing log pipeline usually shows up first as backpressure, then as loss or delay. When outputs stop accepting connections, queues begin to grow, worker threads slow down, and the source side may start dropping events once buffering is exhausted. That matters because the pipeline is not just “slow”; it is no longer preserving the integrity of the event stream.

Security teams often miss the difference between temporary congestion and a real delivery failure. A queue spike may be tolerable if it drains quickly, but repeated growth, stalled destinations, or source drops indicate that logs are no longer reliably reaching the next control point. For monitoring, incident response, and auditability, that is a material failure condition. In practice, many teams only notice pipeline degradation after downstream investigations begin missing the very events they expected to rely on.

For broader control context, NIST SP 800-53 Rev 5 Security and Privacy Controls is useful for thinking about logging, monitoring, and system resilience as control outcomes rather than isolated technical alerts.

How the pipeline degrades before it fully breaks

These logging stacks are designed to absorb short-lived pressure, but they still depend on a chain of assumptions: the source can deliver, the transport can accept, the destination can consume, and buffering can bridge short interruptions. Once any one of those assumptions fails for long enough, symptoms appear in sequence. First, acknowledgement times increase or connections begin to fail. Next, disk or memory queues expand because the system is trying to preserve events locally. If the pressure continues, the pipeline starts shedding work or the source starts reporting drops.

The practical challenge is that not all delay is failure. A slow destination, a network interruption, a credential or socket problem, or a mis-sized queue can all produce similar symptoms. That is why operators should distinguish between latency, buffer growth, and confirmed loss. Latency means the system is struggling. Queue growth means it is buffering risk. Drops mean the risk has already become an incident because data has been discarded.

Unreachable outputs usually indicate a transport or destination issue rather than a source problem.
Rejected socket connections often point to listener misconfiguration, access control, or downstream service failure.
Increasing memory or disk queue usage suggests the pipeline is compensating for sustained backpressure.
Worker latency rising at the same time as queue depth often means the pipeline cannot keep pace with event volume.
Drops reported on the source side are the clearest sign that the system has moved from delay into data loss.

If the only observable issue is transient queue growth that recovers, the pipeline may still be healthy enough for the current load. Once drops, repeated reconnect failure, or persistent queue saturation appear together, the guidance no longer describes normal degradation and the pipeline should be treated as failing.

When normal buffering stops being acceptable

Tighter buffering often improves tolerance for short outages, but it also increases the time between the first fault and the first visible loss, which can make a problem harder to spot. That tradeoff becomes important in environments that treat logs as evidence, not just telemetry. More buffering can buy time, but it cannot compensate for a dead destination, an undersized spool, or a design that never had enough throughput headroom.

There is also an important distinction between a pipeline that is overloaded and one that is misconfigured. A healthy system under burst load may recover after the burst ends. A broken system will keep showing the same symptoms after restarts, reloads, or traffic normalization. Where there is disagreement in the industry, the safest practical rule is to judge by persistence and loss: brief queue growth is an operational condition; repeated drops or permanent backlog are a failure condition.

In mixed environments, syslog-ng and AxoSyslog may behave differently because of destination type, disk buffering settings, transport security, or filter complexity. That is why the same visible symptom should not be assumed to have the same cause. A queue increase caused by a remote collector outage is a different operational problem from a parse failure that prevents messages from being forwarded at all.

Risk and Threat Considerations

A failing log pipeline creates a visibility gap that can hide both operational faults and malicious activity. The main risk is not simply delayed telemetry, but the loss of evidence needed to detect intrusion, reconstruct events, or prove what happened during an incident. If the pipeline silently drops messages, the organisation may believe it has logging coverage when it actually has a blind spot.

Failure mechanism: Attackers and failure conditions both exploit the same weakness: overloaded, unreachable, or misconfigured forwarding paths. When queues saturate or outputs cannot accept data, the system may start discarding events, and that loss can conceal reconnaissance, privilege abuse, or lateral movement if the dropped records were the only trace.

Impact: The immediate consequence is incomplete logging. The downstream consequence is weaker detection, slower triage, and reduced confidence in audits, investigations, and incident timelines.

Standards & Framework Alignment

This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.

MITRE ATT&CK address the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST IR 8596 set the governance and control requirements practitioners need to meet.

Framework	Control / Reference	Relevance
NIST CSF 2.0	PR.PT — Protective Technology	Log forwarding and pipeline resilience support dependable protective telemetry.
Recommendation — Harden log transport and buffering so security telemetry remains available during faults.
CIS Controls v8	8 — Audit Log Management	The question centers on whether logging data is being preserved or lost.
Recommendation — Monitor log paths for drops, backlog, and collector failures to protect audit evidence.
MITRE ATT&CK	T1562 — Impair Defenses	Log pipeline failure or suppression can reduce visibility into attacker activity.
Recommendation — Investigate log loss as a potential defense impairment path when visibility degrades.
NIST IR 8596	IR-4 — Incident Handling	Pipeline failure affects detection, triage, and evidence availability during incidents.
Recommendation — Treat sustained log loss as an incident-impacting condition and preserve alternative evidence.

Practitioner Guidance

What to prioritise: Treat source-side drops as the highest-priority signal because they indicate confirmed loss, not just congestion. Queue growth, worker latency, and reconnect failures are important warning conditions, but they should be escalated differently from evidence that events are no longer being delivered.

What to verify: Confirm whether the problem is downstream unavailability, transport rejection, configuration error, or sustained overload. The useful question is not only “is the pipeline slow?” but “is the pipeline still preserving every event it is expected to carry?”

What good looks like: A healthy pipeline may show short-lived buffering during bursts, but queues should return to baseline and drops should remain absent. If the system can recover only after logs are already lost, it is not meeting its operational purpose.

Practitioner takeaway: In logging pipelines, persistent buffering is a warning, but drops are the threshold that changes the problem from performance degradation to loss of security evidence.