Topic starter
19/08/2026 3:57 pm
TL;DR: During an AWS outage, retry loops can multiply log ingest by 5 to 10 times, and one observed app jumped from 500 lines per second to more than 25,000, according to Sawmills. The practical issue is not just noise, but runaway telemetry cost and delayed detection when systems fail in synchrony.
Editorial analysis by NHI Mgmt Group, based on content published by Sawmills: “When the Cloud Crashes, So Does Your Wallet: How One AWS Outage Could Double Your Observability Bill”.
By the numbers:
- One observed app’s log rate jumped from 500 lines per second to over 25,000.
Key questions
Q: How should teams reduce log storms during cloud outages?
A: Teams should collapse repeated failure messages before ingestion, using fingerprinting, counting, and repeat suppression rules.
Q: Why do retry loops create so much observability cost?
A: Retry loops raise cost because many observability platforms price by volume or message count, so each duplicate error line adds billing even when it adds no new information.
Q: What are the signs that duplicate failure logging is getting out of control?
A: Warning signs include sudden jumps in lines per second, repeated connection-reset or will-retry messages, delayed dashboards, and ingest usage rising faster than the actual business impact of the outage.
Practitioner guidance
- Define retry-aware log aggregation rules Fingerprint repeated or near-identical messages and collapse them into a single enriched event before they reach the paid observability backend.
- Set pre-ingest thresholds for repetitive failures Create policies that switch on repeat suppression when the same error exceeds an agreed count or rate, rather than waiting for manual intervention.
- Test outage conditions against observability cost Simulate dependency failure and measure how quickly log volume, alert noise, and ingestion spend increase under synchronized retries.
Bottom line: Retry storms can turn an ordinary cloud outage into a telemetry amplification event that drains budget and obscures useful signal.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:35 pm
Retry storms are an operational governance problem, not a logging curiosity. When distributed systems fail together, repeated error emissions can consume budget faster than they consume insight. That changes observability from a control plane into a cost amplifier, so practitioners need to treat duplicate suppression as part of resilience design, not an afterthought.
A question worth separating out:
Q: Should teams aggregate logs or tune retry logic first?
A: Teams should do both, but aggregation is usually the fastest way to control cost because it limits damage even when application retries are already deployed. Retry tuning remains important, but it takes longer to roll out across services and does not remove the need for duplicate collapse in the telemetry path.
👉 Read our full editorial: AWS outage retry storms can turn observability into a cost sink