Notifications
Clear all
Topic starter
18/08/2026 2:48 pm
TL;DR: Security data pipelines only deliver what the weakest hop, buffer, and acknowledgement path can sustain, according to Axoflow's analysis of UDP, TCP, HTTP, gRPC, Kafka, OpenTelemetry Collector, and AxoSyslog. Exactly-once is far harder end-to-end than teams assume, so architects need to design for backpressure, crash recovery, and idempotent downstream handling.
NHIMG editorial — based on content published by Axoflow: Message delivery guarantees in security data pipelines
Questions worth separating out
Q: How should security teams design log pipelines when exact delivery cannot be guaranteed?
A: Design for at-least-once delivery, then make every downstream consumer idempotent so retries and replay do not create false actions or duplicate counts.
Q: Why do TCP and buffering still fail to guarantee security telemetry delivery?
A: TCP only confirms that bytes reached the next receive buffer, not that the application processed or forwarded them.
Q: What breaks when a security pipeline uses batch acknowledgements and fan-out?
A: Batch acknowledgements make it hard to know which individual messages were processed before a crash, while fan-out forces the system to track delivery across multiple branches.
Practitioner guidance
- Define the real acknowledgement boundary for every hop Document whether each component gives network-level, application-level, or end-to-end acknowledgement, then compare that with the guarantee your control owner thinks exists.
- Test crash recovery with duplicate and replay scenarios Simulate mid-batch shutdowns, collector restarts, and downstream outages to see whether the pipeline loses events, duplicates them, or stalls.
- Make downstream consumers idempotent by design Assume at-least-once delivery unless you can prove otherwise and build consumers that can safely process repeats without double counting or duplicate actions.
What's in the full article
Axoflow's full blog covers the operational detail this post intentionally leaves for the source:
- Protocol-by-protocol behaviour for UDP, TCP, HTTP, gRPC, Kafka, OpenTelemetry Collector, and AxoSyslog
- Concrete examples of how in-application acknowledgement changes delivery semantics across filters, branches, and crash paths
- The tradeoffs between in-memory and disk-backed buffering when you need to preserve telemetry under failure
- Why exactly-once delivery breaks down in bulk and fan-out designs, even when the transport looks reliable
👉 Read Axoflow's analysis of message delivery guarantees in security pipelines →
Security data pipelines: what delivery guarantees are you really running?
Explore further
18/08/2026 3:30 pm
Message integrity is a governance control, not just a transport feature. Security teams often treat log delivery as an infrastructure problem, but the article shows that delivery semantics define whether evidence is trustworthy at all. If application-level acknowledgement is weak, the organisation may think it has retained telemetry when it has not. That directly affects investigation quality, auditability, and operational resilience. Practitioners should treat pipeline semantics as part of control design, not implementation detail.
A question worth separating out:
Q: Who is accountable when telemetry loss affects investigations or compliance evidence?
A: Accountability usually sits with the team that owns the pipeline design, because delivery guarantees are a control decision rather than a vendor promise. Security and compliance stakeholders should require explicit assurance for acknowledgement, buffering, restart behaviour, and retention. If evidence matters, the control owner must prove that the pipeline preserves it under failure, not just in steady state.
👉 Read our full editorial: Message delivery guarantees in security pipelines still fail at the edges
