Notifications
Clear all
Topic starter
27/08/2026 2:12 pm
TL;DR: Running many operator instances side by side pushed a CPU-bound benchmark from 196k events/sec to 1.20M at parallelism 8, while also cutting queued memory by 65% to 98% across workloads, according to TENZIR. The practical lesson is that pipeline parallelism can remove bottlenecks, but only when planners understand ordering, state, and shuffle costs.
NHIMG editorial — based on content published by TENZIR: Parallel execution in Tenzir pipelines
By the numbers:
- Throughput of the CPU-bound benchmark rose from 196k events/sec on the sequential plan to 1.20M at parallelism 8, a 6.1 times increase.
- At parallelism 8, queued memory fell by 65% to 98% across workloads, with the back-pressured production-like pipeline dropping from 1.95 GiB to 46 MiB.
- The summarize workload reaches 802k events/sec at four instances and 844k at eight before settling back to 826k at 16.
Questions worth separating out
Q: How should security teams tune event pipelines that process IAM or NHI telemetry?
A: Start by separating stateless parsing and enrichment from stateful aggregation or correlation.
Q: Why do some stream-processing pipelines hit a scaling ceiling even when CPU is available?
A: They usually hit a state boundary, not a compute boundary.
Q: What breaks when a pipeline parallelises operators that depend on event order?
A: The main failure mode is semantic drift.
Practitioner guidance
- Profile which operators are safely replicable Map each pipeline stage into stateless, stateful, and order-dependent categories before enabling parallelism.
- Cap partitioning for keyed operators Set explicit limits for shuffles that depend on key affinity, then benchmark the point where extra lanes stop improving throughput.
- Remove buffers that do not add control value Audit internal channels inside replicated lanes and delete buffers that only preserve local sequencing without improving resilience or correctness.
What's in the full article
TENZIR's full analysis covers the implementation detail this post intentionally leaves at the architectural level:
- Operator planner decisions that determine when replication is safe and when ordering must be preserved
- Per-workload benchmark methodology across CPU-bound, summarize, detections, and Suricata-shaped pipelines
- How keyed shuffle caps affect stateful operators and why four partitions became the practical limit
- The internal reorganisation that made future distributed pipeline execution possible
👉 Read TENZIR's analysis of parallel pipeline execution and operator fusion →
Parallel pipeline execution in Tenzir: what changes for practitioners?
Explore further
27/08/2026 2:12 pm
Parallel execution is a control-plane problem, not just a performance setting. The article shows that throughput gains come from planner decisions about replication, fusion, and routing, not from a simple knob that increases worker count. That distinction matters in security telemetry pipelines because the wrong execution model can create queue growth, latency spikes, or uneven processing under load. Practitioners should treat execution planning as part of operational resilience, not merely optimisation.
A question worth separating out:
Q: Should organisations prioritise throughput or memory reduction when redesigning pipelines?
A: The right answer is usually both, but only if the planner can delete useless buffers and preserve the operators that need state. Memory reductions matter most when back-pressure is common, while throughput matters most when the pipeline is CPU-bound. The deciding factor is whether the plan remains correct under burst load and slow sinks.
👉 Read our full editorial: Parallel pipeline execution changes the memory and throughput trade-off
