Join our Newsletter — 33% off our NHI Course

Did AxoSyslog’s flow control starve batching under ACKs?

 

(@nhi-mgmt-group)
Member Moderator
Joined: 2 years ago
Posts: 19717
Topic starter  

TL;DR: AxoSyslog throughput can starve when its batching window is too small for the message flow, as Axoflow explains through in-application ACKs limiting how many messages stay in flight. AxoSyslog then keeps flushing partial batches, which raises latency without an obvious outage.

Editorial analysis by NHI Mgmt Group, based on content published by Axoflow: “AxoSyslog internals: flow control, window size, queues, and batching”.

Key questions

Q: Why does max-connections() affect AxoSyslog throughput even when traffic is steady?

A: Because the source window is divided across connections, a high max-connections() setting can shrink the per-connection share enough that workers cannot gather enough messages to fill batches.

Q: How do security teams know batching is being starved by the window rather than by the source?

A: Look for a pattern where messages arrive, but destination workers repeatedly flush small batches after waiting for batch-timeout().

Q: What breaks when log-iw-size() is too small for batching?

A: Partial batches keep flushing before they fill, so the pipeline spends more time waiting on batch-timeout() than sending efficient batches.

Practitioner guidance

  • Size log-iw-size() against active fan-out Set the source window to cover the number of active connections, destination workers and target batch size, not the maximum possible connection count.
  • Prefer dynamic-window-size() for spiky connection counts Use dynamic-window-size() when max-connections() is much higher than the usual number of active sessions so that idle connection slots do not waste in-flight capacity.
  • Use disk-buffer() when batching is starved Switch to disk-buffer() when partial batches and repeated batch-timeout() flushes show that the memory queue cannot hold enough messages to fill batches.

Bottom line: AxoSyslog can remain operational while still underperforming badly if the message window is too small for its batching design.

Glossary terms: Flow Control, In-application Ack Chain, Disk Buffer, Batch Idle Timeout

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →


This topic was modified 1 day ago by NHI Mgmt Group

   
Quote
(@mr-nhi)
Member Moderator
Joined: 6 months ago
Posts: 20207
 

Throughput collapse in log pipelines is usually a governance problem, not a transport problem. The article shows that a pipeline can remain up while still operating below its expected capacity because windowing and batching are misaligned. That matters because security teams often watch for outages, not for silent under-delivery. The control failure is a sizing assumption that treats throughput as incidental rather than governed.

A question worth separating out:

Q: Should teams use disk-buffer() or tighter window tuning first?

A: Tighten the window first if the goal is efficient batching under memory buffering, because that exposes whether the source window is the real bottleneck. Use disk-buffer() when durability or decoupling matters more than preserving a strict end-to-end ACK chain.

👉 Read our full editorial: AxoSyslog flow control and batching can cap pipeline throughput



   
ReplyQuote
Share:

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.