In August 2021, researchers at cloud security company Wiz found that the Jupyter Notebook feature built into Microsoft's Azure Cosmos DB could be used to reach other customers' databases. By chaining misconfigurations, Nir Ohfeld and Sagi Tzadik escaped their own notebook container and retrieved the primary keys of other Cosmos DB accounts, the long-lived secrets that give full read and write access to a database. Wiz says it gained "complete unrestricted access to the databases of several thousand Microsoft Azure customers", including many Fortune 500 companies. Wiz reported the flaw, which it named ChaosDB, to Microsoft on 12 August. Microsoft disabled the feature within days, paid a $40,000 bounty and on 26 August emailed customers telling them to regenerate their keys, because it could not change them itself. Microsoft says it found no unauthorised access other than the researchers' activity. The case is on our list because live customer credentials were actually obtained, not just exposed in theory.
Key takeaways
- Wiz researchers used a chain of misconfigurations in the Cosmos DB Jupyter Notebook feature to obtain other customers' Cosmos DB primary keys in August 2021.
- A primary read-write key gives full control of a Cosmos DB account. Microsoft could not rotate customers' keys for them, so it asked notified customers to regenerate them.
- Microsoft confirmed to Wiz that "several thousand customers were affected"; Dark Reading reported Wiz's figure of 3,300 customers whose keys were exposed.
- This was coordinated research, not a criminal attack. Microsoft says it found "no unauthorized access other than the researcher activity" and paid Wiz a $40,000 bounty.
- The identity lesson: a static, all-powerful database key is only as safe as every system that can read it, including the cloud provider's own services.
At a glance
| Organisations | Microsoft (Azure Cosmos DB) and the Cosmos DB customers whose keys were reachable, which Wiz says included many Fortune 500 companies |
|---|---|
| When | Research activity 7 to 13 August 2021 according to Microsoft; reported to Microsoft 12 August 2021; disclosed publicly 26 August 2021 |
| Attacker | None known. Found by Wiz researchers Nir Ohfeld and Sagi Tzadik |
| Entry point | The Jupyter Notebook feature in Azure Cosmos DB, enabled by default for new accounts from February 2021 |
| Identities abused | Other customers' Cosmos DB primary read-write keys, plus certificates and secrets inside Microsoft's own infrastructure used to decrypt them |
| Impact | Researchers obtained live database keys for several thousand customers; Microsoft found no other unauthorised access; notified customers had to regenerate their keys |
| Category | NHI. Incident class: confirmed NHI breach (live customer database keys obtained by researchers; no malicious use found) |
What happened
Azure Cosmos DB is Microsoft's managed NoSQL database service. Each account comes with primary and secondary keys that grant access to the data, and applications usually authenticate with them directly. In 2019 Microsoft added a Jupyter Notebook feature so users could query their data from a notebook in the portal. According to Wiz, the feature was turned on automatically for all Cosmos DB accounts from February 2021, so accounts created from then on were potentially exposed.
Wiz's walkthrough, published in November 2021, explains the chain. The researchers escalated privileges inside their notebook container, removed the container's local firewall rules, retrieved certificates from Azure's internal WireServer service and used them to reach a regional Service Fabric cluster. From there they could list other customers' Cosmos DB instances and decrypt stored secrets, including Cosmos DB account keys, notebook authentication tokens and notebook storage keys, for more than 500 instances in that cluster alone. Shir Tamari, Wiz's head of research, told Dark Reading: "We, as Wiz, had access, and we did it with our working laptops."
Wiz says it first exploited the bug on 9 August 2021 and reported it on 12 August. It saw the feature disabled on 14 August, and Microsoft confirmed the behaviour on 16 August, awarded the bounty on 17 August and confirmed on 23 August that several thousand customers were affected. On 26 August Microsoft emailed affected customers. "We have no indication that external entities outside the researcher (Wiz) had access to the primary read-write key," the email said, according to Reuters. Wiz's chief technology officer Ami Luttwak told Reuters: "This is the worst cloud vulnerability you can imagine. It is a long-lasting secret."
Microsoft's own update on 27 August said only customers who used the notebook feature or created accounts during the researchers' activity window were in scope, and that only the primary read-write key was affected. Wiz disagreed on scope. Microsoft notified only customers affected during its roughly one-week research period, Wiz said, while the flaw had probably been exploitable for "at least several months, possibly years", so it urged all Cosmos DB customers to rotate their keys.
Timeline
| Date | Event |
|---|---|
| 2019 | Microsoft adds the Jupyter Notebook feature to Azure Cosmos DB. |
| February 2021 | The notebook feature is turned on by default for Cosmos DB accounts, according to Wiz. |
| 9 August 2021 | Wiz researchers first exploit the flaw and reach other customers' Cosmos DB accounts. |
| 12 August 2021 | Wiz reports the flaw to Microsoft. |
| 14 August 2021 | Wiz observes that the vulnerable feature has been disabled. |
| 17 August 2021 | Microsoft awards Wiz a $40,000 bounty. |
| 23 August 2021 | Microsoft confirms to Wiz that several thousand customers were affected. |
| 26 August 2021 | Wiz publishes ChaosDB; Microsoft emails customers to regenerate their primary keys, as reported by Reuters. |
| 27 August 2021 | Microsoft publishes its update and says it found no unauthorised access beyond the researchers. |
| 11 November 2021 | Wiz publishes the full technical walkthrough. |
How it happened: the identity attack path
- Shared notebook infrastructure. Each customer's notebook ran in a container on infrastructure that also served other customers.
- Container escape. A local privilege escalation and removable firewall rules let the researchers reach internal Azure endpoints from inside the notebook.
- Platform certificates. Azure's WireServer handed out certificates that gave access to a regional Service Fabric cluster.
- Customer secrets decrypted. The cluster held other customers' Cosmos DB keys and notebook tokens in encrypted form, and the certificates the researchers had obtained could decrypt them.
- Full database access. A Cosmos DB primary key grants read, write and delete rights over the account, so holding it meant holding the database.
Impact
- Confirmed: Wiz obtained live primary keys for other customers' Cosmos DB accounts. Microsoft confirmed several thousand customers were affected and asked them to regenerate their primary read-write keys.
- Misuse: none found. Microsoft says its forensic review found no unauthorised access other than the researchers' activity, and that customers not notified have no evidence their primary read-write key was accessed.
- Disputed scope: Wiz said Microsoft notified over 30% of Cosmos DB customers and believed more were at risk because the flaw existed for months; Dark Reading reported Wiz's figure of 3,300 customers. These are Wiz's figures.
- Service: Microsoft turned the notebook feature off for all customers pending a redesign.
What this means for NHI governance
ChaosDB is a story about one kind of non-human identity: the shared, static key. A Cosmos DB primary key carries no user, no scope below the whole account and no expiry. It sits in application configuration for years, and anyone who reads it gets full control. When the cloud provider's own platform could be tricked into decrypting those keys, every customer who relied on them was exposed at once, and only the customers themselves could rotate them. Luttwak's description of it as "a long-lasting secret" is the core of the problem.
The fix Microsoft recommended is the general lesson. Where a service supports identity-based access, such as role-based access control with Azure Active Directory (now Microsoft Entra ID) for Cosmos DB, use it and disable the account keys; where keys remain, rotate them on a schedule and restrict network access. The same theme runs through other Azure cases in our database, including the Microsoft SAS token exposure that Wiz also found. Our Cloud Workload Identity Guide and API Key Management Guide cover the move from shared keys to workload identity.
Recommendations
- Replace database keys with identity-based access. Use Azure Active Directory (now Entra ID) role-based access control for Cosmos DB where supported and disable key-based authentication. See our Cloud Workload Identity Guide.
- Rotate keys on a schedule, and immediately after any provider advisory. Do not wait to be told your key was in scope; regenerate keys whenever a platform flaw could have exposed them. See the Leaked Credential Response Playbook.
- Restrict network reach to data services. Use firewall rules, virtual networks or Private Link so that a stolen key alone is not enough from the internet.
- Turn off preview features you do not use. The notebook feature was enabled by default; review new defaults on cloud services and disable what you do not need.
- Log and review data-plane access. Enable diagnostic logging and alert on access from unfamiliar IP addresses, as Microsoft advised. See our Cloud PAM and CIEM Guide.
- Inventory where cloud keys live. Know which applications hold each key so that rotation does not break production. See our Secrets Management Guide.
Frequently asked questions
What was the ChaosDB vulnerability?
ChaosDB was a chain of flaws in the Jupyter Notebook feature of Azure Cosmos DB, found by Wiz in August 2021. It let a user escape their notebook container, reach internal Azure services and decrypt other customers' Cosmos DB primary keys, which give full access to their databases.
Was ChaosDB exploited by attackers?
Microsoft says its investigation found no unauthorised access other than the Wiz researchers' activity. Wiz did obtain live keys for other customers' accounts, which is why we treat it as a confirmed NHI breach found by researchers rather than a theoretical flaw.
What should Cosmos DB customers have done after ChaosDB?
Regenerate their primary keys, restrict network access with firewalls, virtual networks or Private Link, move to role-based access control where possible, enable logging and review access from unusual IP addresses. Wiz advised all Cosmos DB customers to rotate keys, not only those Microsoft notified.
Related NHI Mgmt Group resources
Microsoft SAS Token Exposure 2023 · Azure Key Vault Contributor Escalation 2024 · Entra ID Actor Token Flaw 2025 · Cloud Workload Identity Guide · API Key Management Guide
How NHI Mgmt Group can help
Shared database and storage keys are some of the most powerful and least governed credentials in cloud estates. We help organisations find them, map which workloads depend on them and move to short-lived, identity-based access. See our NHI and AI agent security training.
References
- Wiz: ChaosDB: How we hacked thousands of Azure customers' databases (26 August 2021)
- Reuters via CNBC: Microsoft warns thousands of cloud customers of exposed databases, email shows (26 August 2021)
- Microsoft Security Response Center: Update on Vulnerability in the Azure Cosmos DB Jupyter Notebook Feature (27 August 2021)
- Dark Reading: Microsoft Azure Cloud Vulnerability Exposed Thousands of Databases (27 August 2021)
- CERT-EU: Security Advisory 2021-045, Microsoft Cosmos DB Vulnerability (31 August 2021)
- Wiz: ChaosDB explained: Azure's Cosmos DB vulnerability walkthrough (11 November 2021)