Security Chaos Engineering: Testing Controls Under Failure

Security controls are usually tested when they work, and fail in production when something they depend on stops working. A secret store that becomes unreachable, a log forwarder that dies, a firewall rule changed by a script or a certificate rotated early can each turn a sound design into an outage or a blind spot. Security chaos engineering applies the chaos engineering method (steady state, hypothesis, controlled fault, measurement) to those failure modes, so you learn how controls and recovery behave before an incident or an attacker teaches you. This guide covers experiment design, the fault injection features in AWS Fault Injection Service and Azure Chaos Studio, the guardrails and identities that keep experiments safe, and how to measure results. It is for security engineers, SRE and platform teams, and detection engineers who own resilience of security tooling.

Key takeaways

Test the failure, not the feature. The question is what happens to authentication, secrets access, logging and alerting when a dependency fails, and whether the system fails safe or fails open.
Every experiment needs a hypothesis and a stop condition. AWS frames it as a steady state metric, a hypothesis about mitigating controls, and a guardrail alarm that halts the fault; Azure gates it with roles, an experiment identity and onboarded targets.
Some faults do not undo themselves. Azure's Key Vault Increment Certificate Version fault is not reverted, and stopping EC2 with skipOsShutdown can lose data; read the action reference before each run.
The chaos platform is a privileged identity. Experiment roles and managed identities can isolate subnets, deny vault access and stop workloads, so scope and monitor them like any administrator.
Results belong in detection and recovery engineering. Each experiment should end with a measured outcome: alert fired or not, time to detect, time to recover, and a tracked fix.

What it is and how it differs from other testing

AWS's Well-Architected best practice REL12-BP04 defines chaos engineering, quoting the Principles of Chaos Engineering, as experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. The security variant asks the same question of controls and security-relevant dependencies.

Practice	What it injects	Question answered	Typical owner
Security chaos engineering	Infrastructure and dependency failures: network loss, vault denial, certificate change, API errors, instance stop	Do controls fail safe, keep logging and recover within target?	Platform security, SRE
Breach and attack simulation	Simulated attacker techniques	Do controls block or detect known techniques?	Security operations; see the Breach and Attack Simulation (BAS) Guide
Purple teaming	Human-led adversary emulation with defenders	Can we detect and respond to this actor's behaviour?	Red and blue teams; see the Purple Teaming Guide
Disaster recovery test	Loss of a site, Region or system	Can we restore service within RTO and RPO?	Resilience and IT operations
Tabletop exercise	A scenario discussed, nothing injected	Do people and decisions work?	Incident response; see the Tabletop Exercise Guide

Designing a security experiment

Choose the fault from evidence: AWS recommends mapping each resilience design element to the faults it mitigates, using critical dependencies, past incidents and failure mode analysis, then prioritising by frequency and impact.
Define steady state: a measurable normal, for example sign-in success rate, secret retrieval latency, events per minute arriving in the SIEM, or WAF block rate.
Write the hypothesis: AWS's template is "If specific fault occurs, the workload will mitigating controls to maintain metric impact". A security example: "If the application loses access to Key Vault for ten minutes, it will serve cached secrets, deny new privileged operations, and an alert will reach the on-call analyst within five minutes."
Set guardrails: stop conditions, blast radius (tags, a single subnet, a percentage of calls), duration, and a named person who can halt it.
Inject, observe, compare: run the fault, record metrics and alerts, compare against the hypothesis.
Fix and repeat: if steady state was not held or detection did not fire, change the design or rule and rerun. Validated experiments become regression tests in the pipeline and inputs to game days (REL12-BP05).

Security experiments worth running

Control under test	Fault	Platform feature	What good looks like
Secrets access fail-safe behaviour	Vault unreachable	Azure Key Vault Deny Access (urn:csci:microsoft:keyVault:denyAccess/1.0); AWS aws:network:disrupt-vpc-endpoint on the Secrets Manager endpoint	Application degrades without falling back to hard-coded or cached plaintext credentials on disk; alert fires
Certificate handling	Certificate disabled or rotated	Azure Key Vault Disable Certificate (re-enabled at end); Increment Certificate Version (not reverted)	Clients pick up the new version without outage; pinning failures are caught; see the Certificate Expiry Outages and 47-Day TLS Certificates Guide
Network segmentation and detection	New deny or allow rule applied	Azure NSG Security Rule fault; AWS aws:network:disrupt-connectivity with scope prefix-list or vpc	Dependent services fail closed; network monitoring sees the change (compare ATT&CK T1686 Disable or Modify System Firewall)
Log pipeline resilience	Forwarder host or task stopped, collector endpoint blocked	AWS aws:ecs:stop-task, aws:ec2:stop-instances, aws:network:disrupt-connectivity; Azure Network Disconnect agent fault	A "no data" detection fires within its window; buffered events arrive after recovery (compare T1685 Disable or Modify Tools)
Authentication dependency	DNS failure for the identity provider host	Azure DNS Failure agent fault (Windows only)	Applications deny access rather than bypass; break-glass process works
Cloud API error handling in security automation	Throttling or unavailable errors on a role's API calls	AWS aws:fis:inject-api-throttle-error and aws:fis:inject-api-unavailable-error (service ec2 or kinesis)	Remediation or isolation scripts retry correctly and report failure, not false success
Backup replication	Replication paused	AWS aws:s3:bucket-pause-replication	Replication lag alert fires; RPO stays within target

Detection-side experiments pair naturally with the Detection Engineering Guide: a fault that should produce an alert is a test case for the rule.

Guardrails and safe operation

AWS Fault Injection Service

Experiment templates: hold actions, targets (specific resources or tag-based selection) and stop conditions. A stop condition is a CloudWatch alarm; AWS's Well-Architected guidance notes up to five per template.
Safety lever: one per account per Region. Engaging it with aws fis update-safety-lever-state --id "default" --state "status=engaged,reason=..." stops running experiments and cancels any that try to start; stopped experiments cannot be resumed. Check with aws fis get-safety-lever --id "default".
Experiment role: trust only fis.amazonaws.com, add aws:SourceAccount and aws:SourceArn conditions against the confused deputy problem, and scope permissions to resource ARNs or tags rather than starting from the broad AWSFaultInjectionSimulator managed policies.
Know which actions restore: aws:network:disrupt-connectivity swaps in a cloned network ACL tagged managedByFIS=true and restores the original when the action completes; skipOsShutdown on stop or terminate bypasses graceful shutdown and may cause data loss.

Azure Chaos Studio

Three gates: Resource Manager permission to start (Microsoft.Chaos/experiments/start/action), an experiment managed identity with rights on each target, and target plus capability onboarding. If any gate is missing the fault does not run.
Roles: Chaos Studio Experiment Contributor, Chaos Studio Operator (run, not create), Chaos Studio Reader and Chaos Studio Target Contributor. Microsoft notes the Contributor role's */write includes experiment write, so restrict Contributor where experiments live.
Workspaces: Microsoft states Experiments (classic) is being replaced by Chaos Studio Workspaces; build new programmes there.
Fault caveats: NSG deny rules do not break existing connections until they have been idle for four minutes; agent network faults affect new connections only; any external change to the NSG rule during the action fails the experiment.
Data: do not place secrets in fault parameters; Customer Lockbox can gate Microsoft access to experiment data.

Rolling out a programme

Read-only discovery: list security dependencies per critical service (identity provider, vault, KMS, log path, WAF, DNS) and the alert each failure should produce. Many gaps appear at this step before any fault is injected.
Platform setup: create a dedicated experiment role or managed identity per environment, tag-scope targets, configure stop condition alarms and document the safety lever or cancel procedure.
Pre-production first: AWS recommends planning and testing experiments outside production because the service performs real actions on real resources. Run each experiment there at least once.
Production pilot: one low-risk fault with a short duration (minutes), small blast radius, the owning team informed and a person watching. Disruptive: faults affect real users within the blast radius.
Rollback: for continuous faults, cancel the experiment or engage the safety lever; for non-reverting faults (certificate version increment, instance termination) have the manual restore written down before you start.
Automate validated experiments: schedule them or run in CI/CD as regression tests, and capture time, conditions and observations for trend analysis, as AWS recommends.

Decision guidance

Small teams: start with two or three experiments on your most critical dependency (usually the identity provider or secrets store), in pre-production, run quarterly. Native cloud services avoid extra agents.
Large or regulated estates: run a catalogue of experiments per service tier, link each to a control ID and detection rule, and feed results to operational resilience reporting.
Native service or open-source tool: AWS FIS and Azure Chaos Studio integrate with cloud IAM, logging and stop conditions. Kubernetes-heavy estates may also use Chaos Mesh or Litmus, which AWS FIS can trigger through aws:eks:inject-kubernetes-custom-resource using Kubernetes RBAC instead of IAM.
Production or not: production gives real answers; pre-production gives safety. Move a given experiment to production only when it has passed in pre-production and its stop conditions have been seen to work.

Detection and measurement

Audit the chaos platform itself: experiment start, stop and template changes (CloudTrail for FIS API calls, Azure Activity Log for Microsoft.Chaos operations) should be alerted on outside approved windows. An unauthorised experiment is an attack.
Correlate experiments with alerts: tag experiment runs so the SOC can confirm which alerts were expected and which never fired.

Metrics:

Percentage of critical security dependencies with at least one validated experiment.
Hypotheses held versus failed per quarter, with fixes closed.
Time to detect each injected security failure versus target.
Time to recover steady state after the fault ends.
Number of experiments halted by stop conditions (shows guardrails work, and where blast radius was misjudged).

Common failure patterns

No hypothesis: faults are injected "to see what happens", nobody agreed what success looked like, and nothing is fixed.
Stop conditions that never trigger: alarms watch the wrong metric or have periods longer than the experiment.
Over-privileged experiment identity: a FIS role with wildcard resources or a Chaos Studio managed identity with Contributor on the subscription becomes a standing path to disrupt production.
Assuming everything reverts: a certificate version increment or terminated instance is permanent, and the team learns this after the run.
Testing only availability: the service stays up because it silently fell back to an insecure path, such as skipping token validation, which nobody measured.
Experiments outside change control: the SOC treats an experiment as an incident, or worse, treats a real incident as an experiment.

Where identity fits

Identity dependencies are prime experiment targets: what happens when workloads cannot reach the identity provider, the vault or the token service, and do they fail closed? The chaos tooling also introduces new non-human identities (FIS experiment roles, Chaos Studio managed identities, agent identities) that hold destructive permissions. Inventory them, scope them to tagged targets, and review them like other privileged workload identities. See the Detection Engineering for Non-Human Identities Guide.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev 5	SI-6 Security and Privacy Function Verification	Verifying security functions operate correctly, including under failure
NIST SP 800-53 Rev 5	CA-2 Control Assessments; CP-4 Contingency Plan Testing; IR-3 Incident Response Testing	Experiments as evidence of control and recovery testing
NIST CSF 2.0	ID.IM-02; DE.CM-01; RC.RP-05	Improvements from tests, monitoring verified, restored state confirmed
AWS Well-Architected	REL12-BP04; REL12-BP05	Chaos engineering and game days
MITRE ATT&CK v20	T1685 Disable or Modify Tools; T1686 Disable or Modify System Firewall	Failures that mimic defence impairment, testing whether detection notices

Practitioner checklist

List security dependencies per critical service and the alert each failure should raise.
Write a steady state and hypothesis for each experiment before building it.
Create scoped experiment identities with aws:SourceAccount and aws:SourceArn conditions or least-privilege managed identities.
Configure stop conditions and confirm they fire in pre-production.
Document the safety lever or cancel procedure and who may use it.
Check the action reference for faults that do not revert, and write the manual restore first.
Alert on chaos platform activity outside approved windows.
Record results, track fixes and rerun failed hypotheses.
Promote validated experiments to scheduled or pipeline regression tests.

Security chaos engineering: prove controls survive failure with safe, measured experiments