Notifications
Clear all
Topic starter
19/08/2026 3:57 pm
TL;DR: During an AWS outage, retry loops can multiply log ingest by 5 to 10 times, and one observed app jumped from 500 lines per second to more than 25,000, according to Sawmills. The practical issue is not just noise, but runaway telemetry cost and delayed detection when systems fail in synchrony.
NHIMG editorial — based on content published by Sawmills: When the Cloud Crashes, So Does Your Wallet: How One AWS Outage Could Double Your Observability
By the numbers:
- During the outage, one observed app’s log rate jumped from 500 lines per second to over 25,000, a 50x spike.
Questions worth separating out
Q: How should security teams handle retry storms in cloud observability pipelines?
A: Treat retry storms as a pipeline design issue, not just an application bug.
Q: Why do AWS outages create such large logging and telemetry spikes?
A: Because many distributed services retry simultaneously when shared dependencies degrade.
Q: How do you know if log aggregation is actually working?
A: You know it is working when key identity and cloud events appear in the same schema, can be queried across sources, and support a complete investigation without manual data stitching.
Practitioner guidance
- Implement pre-ingest log aggregation Collapse repeated retry and timeout messages at the collector or forwarder so only the first event and repeat count are shipped to the backend.
- Set alerts on retry-driven ingest spikes Create thresholds for abnormal log volume growth tied to known failure classes such as connection resets, timeout loops, and upstream dependency errors.
- Tune client retry behaviour Review retry backoff, jitter, and circuit breaker settings for AWS-dependent services so outages do not synchronise retries across the estate.
What's in the full article
Sawmills' full analysis covers the operational detail this post intentionally leaves for the source:
- Message fingerprinting patterns for collapsing repeated AWS failure logs before ingestion
- Collector and forwarder placement options for OpenTelemetry, Vector, and similar pipelines
- Cost-control heuristics for identifying when retry volume becomes an observability incident
- Implementation detail on how to preserve counts and context while discarding duplicate events
👉 Read Sawmills' analysis of AWS outage retry storms and log cost control →
AWS retry storms and observability bills: what should teams change?
Explore further
19/08/2026 4:35 pm
Retry storms are an observability control problem, not just a cloud outage symptom. The article shows how distributed retry behaviour can convert a short-lived AWS disruption into a telemetry flood. That matters because the real failure is governance over repeated events, not only service availability. Teams that treat logs as raw exhaust will overpay for duplicates and still miss the signal. Practitioners should design for suppression, not just collection.
A question worth separating out:
Q: What should teams do when observability costs rise during an outage?
A: Validate whether the spike is caused by retry loops, then apply suppression at the collector, reduce retry synchronisation, and cap ingestion for known noisy classes. If the same incident also depends on shared service credentials, review the workload identities behind the traffic and narrow their permissions at the same time.
👉 Read our full editorial: AWS outage retry storms can turn observability into a cost sink
