Cloudflare Breach

NHI Mgmt Group

Overview

On Thanksgiving Day, November 23, 2023, Cloudflare disclosed a significant breach involving their internal Atlassian systems. The intrusion occurred after attackers used credentials stolen during the October 2023 Okta breach. This breach, attributed to a suspected nation-state actor, targeted Cloudflare’s Confluence wiki, Jira bug database, and Bitbucket source code repository. The attackers initiated access on November 14, 2023, and maintained a presence until November 23, when the activity was detected and access terminated.

Origin: The Okta Breach

The breach’s origins trace back to the compromise of an Okta customer support engineer’s account in October 2023. This earlier attack allowed threat actors to view customer credentials, which were then exploited to target other organizations like Cloudflare. The incident highlights the cascading risks posed by third-party security breaches.

Incident Timeline

October 18, 2023

Cloudflare experienced a second breach due to a compromise in Okta’s systems, where a set of credentials were leaked. Despite thousands of credentials being rotated, four key credentials were missed: one service token from Moveworks, a service account for Smartsheet with administrative access to Jira, a Bitbucket service account, and an AWS account with no access to sensitive data. These credentials were mistakenly believed to be unused, but this oversight allowed the attacker to access Cloudflare's Atlassian products and maintain persistence. The breach was not due to flaws in the third-party systems, but rather to the failure to rotate these specific credentials

November 14, 2023

The attacker started looking into Cloudflare's systems on November 14 using the compromised credentials. They attempted to login to Okta and the Cloudflare Dashboard but were denied access.
They accessed a segmented AWS environment used for the Cloudflare Apps marketplace, which did not have access to customer data or the global network. The service account used to access this environment was revoked, and its integrity was validated.

November 15, 2023

On November 15, the threat actor successfully accessed Cloudflare’s Atlassian services (Jira and Confluence) using the Moveworks service token and the Smartsheet service account. They accessed Jira tickets and wiki pages, searching for sensitive information like remote access details and secrets.
They accessed 36 Jira tickets and 202 wiki pages, mostly related to internal configurations, vulnerability management, and Cloudflare’s response to the Okta incident.

November 16, 2023

Using the Smartsheet account, the attacker created a new user account within Atlassian, making it appear as a legitimate Cloudflare user. They added this account to several groups to ensure persistent access.

November 17-20, 2023

The attacker paused their activity, though they briefly tested access during this time. No significant actions were taken during this period.

November 22, 2023

On November 22, the attacker used the Smartsheet service account’s administrative access to install Sliver, a tool used for command and control (C2). They used this to maintain persistent access via the ScriptRunner for Jira plugin.
The attacker attempted lateral movement but failed to access a non-production server in São Paulo due to misconfigured access controls. They also viewed and downloaded 76 source code repositories from Bitbucket, though it was unclear if any code was exfiltrated.

November 23, 2023

At 16:00, Cloudflare’s security team was alerted to the threat actor’s activity and immediately began investigating. The Smartsheet service account was deactivated by 16:35, and the user account created by the attacker was disabled by 17:23.
The incident was formally declared at 17:43, and firewall rules were implemented to block the attacker’s known IP addresses.

November 24, 2023

By 10:44, the last known activity from the attacker was logged, and by 11:59, Sliver was removed from the system, cutting off all access.
Throughout the investigation, it was confirmed that the attacker had not accessed Cloudflare’s global network, data centers, SSL keys, customer databases, or any sensitive infrastructure. Their activity was confined to the Atlassian suite and a single server.

Cloudflare’s Response

Cloudflare acted swiftly upon discovering the breach:

Access was severed within 24 hours of detection.
Over 5,000 credentials were rotated across production systems.
A forensic investigation of nearly 4,900 systems was conducted.
All Atlassian servers and global systems affected were re-imaged and rebooted.
Equipment in the São Paulo data center was returned to manufacturers for security verification.

Lessons Learned

Secrets Management: Unrotated credentials were a key vulnerability. Tools that monitor and manage secrets sprawl could mitigate such risks.
Supply Chain Security: Third-party platforms like Okta can become critical attack vectors. Regular audits and strict access controls are essential.
Early Detection: While Cloudflare detected the breach within nine days, a relatively short dwell time, earlier detection mechanisms, like honeytokens or enhanced logging, could have further limited damage.

Conclusion

While Cloudflare’s operations and customer data remained secure, the breach is a sobering example of how one compromised system can have far-reaching consequences.

The incident highlights the necessity of strong, proactive cybersecurity measures and cross-industry collaboration to prevent similar events in the future.