Look for a pattern where messages arrive, but destination workers repeatedly flush small batches after waiting for batch-timeout(). That usually means the pipeline has enough traffic overall but not enough in-flight messages per worker to build full batches.
How to tell it is a batch-window bottleneck, not a source bottleneck
The signal is throughput with poor aggregation, not starvation at the origin. If events keep arriving but each worker still emits tiny batches only after the timeout expires, the source is supplying work, yet the per-worker queue is not filling fast enough to trigger size-based flushes. That usually points to windowing, partitioning, or worker fan-out rather than source silence.
The practical test is to separate arrival rate from batch-fill rate. A healthy source can still look “slow” downstream when messages are spread across too many workers, shards, or keys, because each consumer sees too little traffic to assemble a full batch before the timer fires. In that case, the timeout is doing the flushing, not the source.
The easiest confirmation is to compare observed batch size against the configured batch limit and flush interval. If the workers repeatedly flush at small sizes while latency stays close to the timeout threshold, the batching window is constraining throughput. If the source were the limiting factor, you would expect gaps in arrivals and long idle periods, not a steady stream of messages that fail to coalesce.
What the worker pattern is really telling you
A worker that repeatedly times out with partial batches is usually operating below its fill threshold. That can happen when input volume is too low, but it can also happen when volume is adequate overall and simply fragmented across too many independent consumers. The distinction matters because the remedy is different: tune the window, rebalance partitioning, or reduce worker fan-out rather than chasing the upstream source.
In distributed pipelines, this often shows up as uneven per-worker utilization. One queue may be active, but no single worker gets enough in-flight messages to build a full batch. The result is latency dominated by the timer, not by upstream scarcity. Watching worker-local queue depth, not just global ingest rate, is the key diagnostic.
A useful mental model is that batch systems need both supply and concentration. A source can be busy overall while still underfeeding each destination shard. When that happens, the system is not source-starved, it is window-starved, meaning the batch policy is waiting for a density of work that never arrives in one place.
How security and operations teams should validate the cause
Start by checking whether total inbound volume is stable across the measurement window. Then compare that against per-worker batch composition, timeout frequency, and queue depth. If the source is healthy but batches remain small, the likely issue is routing shape, worker count, or batch parameters, not upstream production.
For teams operating this kind of pipeline, the strongest evidence is a repeating pattern: messages arrive continuously, but flushes happen on timeout with little variation in batch size. That pattern is the opposite of source starvation. It shows the system has work, but the batching window is too short or too thinly distributed to accumulate full batches before expiry.
When the diagnosis is uncertain, reduce one variable at a time. Lower the worker count, widen the window, or temporarily concentrate traffic to a single consumer group. If batch sizes rise immediately, you have confirmed that the source was not the constraint. If they do not, then the source or upstream routing deserves a deeper look.
Practitioner Guidance
What to verify: Compare global ingest rate with per-worker queue depth and flush reason. If the source rate is healthy but flushes are timeout-driven, treat the batching window as the primary constraint.
Decision rule: If batches grow when you reduce worker fan-out or widen the timeout, the bottleneck is the window or partitioning shape. If batches stay small even after concentration, investigate the source path and upstream delivery rate.
Practitioner takeaway: Do not judge batching health by source traffic alone, judge it by whether a single worker can accumulate enough in-flight messages to beat the timeout and flush on size instead of on the clock.
Related resources from NHI Mgmt Group
- How do security teams know browser hijacking is happening rather than ordinary user activity?
- How do security teams know if a build environment is leaking source during compilation?
- How do security teams know whether multi-source vulnerability tracking is working?
- How do security and platform teams know whether a connection pool is failing because of app design rather than database capacity?