On 26 March 2024, Oligo Security disclosed ShadowRay, a campaign in which attackers had been taking over internet-exposed clusters running Ray, the open source framework for scaling AI and Python workloads maintained by Anyscale. The attackers used CVE-2023-48022, the lack of authentication on Ray's Jobs API, to run their own code on clusters. Oligo says the earliest evidence dates from 5 September 2023. On compromised clusters, Oligo found OpenAI, Hugging Face, Stripe and Slack tokens, AWS, Google Cloud and Azure access, production database credentials and private SSH keys, alongside cryptocurrency miners and reverse shells. Anyscale disputes that CVE-2023-48022 is a vulnerability at all. It says Ray is designed to execute arbitrary code and must run only inside a trusted network. It told customers of its managed service they were not affected and released tools to check for exposed ports. Oligo called it the first known attack campaign targeting AI workloads.
Key takeaways
- Oligo Security says attackers exploited CVE-2023-48022 on publicly exposed Ray clusters for about seven months before its 26 March 2024 disclosure, with the earliest evidence on 5 September 2023.
- Compromised clusters exposed OpenAI, Hugging Face, Stripe and Slack tokens, cloud access to AWS, Google Cloud and Azure, database credentials and SSH keys, according to Oligo. Many clusters ran as root.
- Oligo describes "hundreds of compromised clusters" in the body of its report and "thousands of publicly exposed Ray servers" in its summary, and values the compromised machines at almost $1 billion. These are the vendor's own figures and estimates.
- Anyscale disputes the CVE, calling missing authentication "not in our opinion a vulnerability, or even a bug", and says Ray must only run on trusted networks. It said it had no reports of malicious activity from its customers.
- The identity lesson: AI compute clusters are full of non-human credentials, and an unauthenticated control plane hands all of them to anyone who can reach it.
At a glance
| Organisations | Organisations running self-managed Ray clusters exposed to the internet, in sectors including education, cryptocurrency, biopharma, medical and video analytics, according to Oligo; Anyscale, Ray's maintainer |
|---|---|
| When | Exploitation from at least 5 September 2023, according to Oligo; disclosed 26 March 2024 |
| Attacker | Unknown. Multiple actors deployed miners and reverse shells; no attribution published |
| Entry point | Ray's unauthenticated Jobs API on the dashboard port (8265 by default), exposed to the internet (CVE-2023-48022, disputed) |
| Identities abused | Cloud credentials for AWS, Google Cloud, Azure and Lambda Labs; OpenAI, Hugging Face, Stripe and Slack tokens; database credentials; SSH keys; Kubernetes API access |
| Impact | Remote code execution, credential exposure and compute hijacked for cryptomining on hundreds of clusters, according to Oligo; Anyscale managed-service customers not affected |
| Category | NHI, LLM / AI platform. Incident class: confirmed NHI breach (exploited in the wild on AI clusters holding cloud and AI API credentials; flaw disputed by the vendor) |
What happened
Ray is an open source framework for distributing Python and AI workloads, such as model training, fine-tuning and inference, across clusters of machines. Clusters often run on expensive GPU instances and hold the credentials that AI work needs: cloud keys to read datasets and write models, API tokens for model providers and services, and database access. In August 2023 Bishop Fox reported several Ray flaws, according to The Hacker News. Anyscale fixed four of them in Ray 2.8.1 in November 2023 but did not change the fifth, CVE-2023-48022, which lets anyone with network access to the Ray dashboard submit jobs that run arbitrary code. MITRE scored it 9.8 and marked it disputed.
Anyscale explained its position in November 2023. Because Ray is built to run arbitrary code, the company said, a user who can send requests to the cluster already has that power, so missing authentication is "not in our opinion a vulnerability, or even a bug". Ray's security boundary sits outside the cluster, and untrusted machines, with "the public internet" as Anyscale's example, must be kept away. It added that "reasonable minds can differ" and promised authentication as a defence-in-depth feature.
On 26 March 2024, Oligo's researchers Avi Lumelsky, Gal Elbaz and Guy Kaplan reported that attackers had been doing exactly what the CVE describes, against clusters left open to the internet. Oligo traced the earliest activity to 5 September 2023 and saw the first cryptominer installed on 21 February 2024. On compromised clusters it found XMRig, NBMiner and Zephyr miners, reverse shells, and access to a wide range of secrets: OpenAI, Hugging Face, Stripe and Slack tokens, AWS, Google Cloud, Azure and Lambda Labs access, production database credentials, private SSH keys and password hashes. In one case a KubeRay operator ran with administrator rights over Kubernetes. "When attackers get their hands on a Ray production cluster, it is a jackpot," the researchers wrote, as quoted by The Hacker News.
Anyscale responded that it had "notified all Anyscale customers of the vulnerability and that they are not affected," according to TechTarget. It said it had received no customer reports of malicious activity linked to the flaw, released a Ray Open Ports Checker, and planned authentication and default checks for Ray 2.11. Oligo told TechTarget it had seen about 500 exposed IP addresses, possibly representing hundreds of organisations.
Timeline
| Date | Event |
|---|---|
| August 2023 | Bishop Fox reports Ray vulnerabilities, including the missing authentication on the Jobs API, according to The Hacker News. |
| 5 September 2023 | Earliest evidence of exploitation found by Oligo. |
| 30 November 2023 | Anyscale publishes its update on the Ray CVEs, disputing CVE-2023-48022. |
| 21 February 2024 | First cryptominer installation observed by Oligo on a compromised cluster. |
| 26 March 2024 | Oligo Security discloses the ShadowRay campaign. |
| 27 March 2024 | TechTarget and The Hacker News report Anyscale's response: managed-service customers not affected, and a new tool to check for open ports. |
How it happened: the identity attack path
- Find exposed clusters. Attackers located Ray dashboards reachable from the internet, typically on port 8265.
- Submit a job, no identity needed. The Jobs API accepted work from anyone who could reach it, so attackers ran their own code on the cluster, often as root.
- Harvest the cluster's identities. Environment variables, files and configured services gave up cloud credentials, model provider and SaaS tokens, database passwords and SSH keys.
- Move outward. Cloud and Kubernetes credentials, including an operator with administrator rights in one case, offered paths beyond the Ray cluster itself.
- Monetise the GPUs. Attackers installed cryptominers and reverse shells to keep using the compute.
Impact
- Confirmed by researchers: active exploitation of exposed Ray clusters, cryptominers and reverse shells, and access to cloud, AI API, SaaS and database credentials on those clusters, according to Oligo.
- Vendor estimates: Oligo values the compromised machines at almost $1 billion, based on cloud GPU pricing. Its count of affected clusters ranges from "hundreds" to "thousands" in different parts of its report.
- Not affected: customers of Anyscale's managed platform, according to Anyscale, which also said it had no customer reports of malicious activity.
- Potential: stolen cloud keys and AI API tokens can be reused long after the cluster is cleaned up, for example for LLMjacking or data theft, unless they are rotated.
What this means for NHI and AI agent security
ShadowRay belongs on an NHI list because the prize was not only GPU time. AI clusters gather the non-human identities that AI work depends on: cloud keys, model provider tokens, Hugging Face tokens, database credentials and Kubernetes service accounts. With no authentication on the control plane, every one of those identities was one HTTP request away from anyone who found the cluster. Keys like these can feed attacks such as LLMjacking, where attackers run models on someone else's account.
The dispute over the CVE matters less than the deployment reality. Whether missing authentication is a bug or a design choice, the clusters that were attacked were reachable from the internet and held long-lived secrets. The fix is the same either way: put AI infrastructure behind network controls and authentication, give workloads short-lived identities rather than static keys and assume anything on an exposed node is compromised. See our AI Infrastructure Workload Identity Guide and LLMjacking Guide.
Recommendations
- Take AI control planes off the internet. Keep Ray dashboards and job APIs inside trusted networks, restricted by firewall rules or security groups, as both Anyscale and Oligo advise. See our AI Infrastructure Workload Identity Guide.
- Add authentication in front of the cluster. Use any authentication your Ray version offers, or put an authenticating proxy in front of the dashboard port.
- Rotate every credential on exposed clusters. Treat cloud keys, model provider and SaaS tokens, database passwords and SSH keys on any exposed node as stolen. See our Leaked Credential Response Playbook.
- Use short-lived workload identities. Give clusters cloud roles and federated tokens instead of static access keys in environment variables. See our Cloud Workload Identity Guide.
- Scope AI API keys and watch their use. Limit model provider keys by project and spend, and alert on unusual usage. See our LLMjacking Guide.
- Do not run AI workloads as root or with cluster-admin. Limit the privileges of jobs, operators and Kubernetes service accounts.
Frequently asked questions
What is ShadowRay?
ShadowRay is Oligo Security's name for a campaign, disclosed on 26 March 2024, in which attackers exploited the unauthenticated Jobs API of internet-exposed Ray clusters (CVE-2023-48022) to run code, steal credentials and mine cryptocurrency. Oligo says it began at least as early as 5 September 2023.
Is CVE-2023-48022 a real vulnerability?
It is disputed. Anyscale says Ray is designed to run arbitrary code and must only be deployed in a trusted network, so missing authentication is not a bug. Oligo and others say clusters were exploited in the wild because of it. Either way, Ray clusters exposed to the internet were taken over.
What data was exposed in ShadowRay?
According to Oligo, compromised clusters held OpenAI, Hugging Face, Stripe and Slack tokens, AWS, Google Cloud, Azure and Lambda Labs access, production database credentials, private SSH keys, password hashes and AI models and datasets.
Related NHI Mgmt Group resources
LLMjacking 2024 to 2026 · Hugging Face Spaces breach 2024 · Kubeflow cryptomining attacks 2020 · AI Infrastructure Workload Identity Guide · LLMjacking Guide
How NHI Mgmt Group can help
AI infrastructure concentrates cloud keys, model provider tokens and service accounts in a few powerful clusters. We help teams find those identities, move them to short-lived credentials and keep AI control planes out of reach. See our NHI and AI agent security training.
References
- Anyscale: Update on Ray CVEs CVE-2023-6019, CVE-2023-6020, CVE-2023-6021, CVE-2023-48022, CVE-2023-48023 (30 November 2023)
- Oligo Security: ShadowRay: First Known Attack Campaign Targeting AI Workloads Actively Exploited In The Wild (26 March 2024)
- The Hacker News: Critical Unpatched Ray AI Platform Vulnerability Exploited for Cryptocurrency Mining (27 March 2024)
- TechTarget: Unpatched flaw in Anyscale's Ray AI framework under attack (27 March 2024)