On 10 June 2020, Microsoft warned that attackers were hijacking Kubernetes clusters that run Kubeflow, the open-source machine learning toolkit, to mine cryptocurrency. Azure Security Center first saw the activity in April 2020, when a suspicious image containing the XMRig miner appeared on many different clusters. Microsoft said the attack affected tens of Kubernetes clusters, most of them running Kubeflow. The way in was a dashboard that some teams had exposed to the internet by switching Kubeflow's Istio ingress gateway to a public load balancer. From the dashboard, attackers could start a Jupyter notebook server with their own image, or run code inside a notebook and use the Kubernetes service account mounted into it, which by default can deploy containers in its namespace. Microsoft wrote that "The attacker managed to move laterally and deploy the container using the mounted service account." No victim organisations were named, and Microsoft published checks and mitigations for Kubeflow users.
Key takeaways
- Microsoft's Azure Security Center saw a cryptomining image deployed across tens of Kubernetes clusters from April 2020, most of them running Kubeflow, and disclosed the campaign on 10 June 2020.
- Initial access came through Kubeflow dashboards exposed to the internet, a change some users made for convenience, according to Microsoft.
- The workload identity did the rest: a service account mounted into each notebook server can, by default, deploy containers in its namespace, and Microsoft says the attacker used it to deploy the miner.
- Machine learning clusters were attractive because their nodes are often powerful and sometimes include GPUs; Microsoft called this the first attack it had seen that targets Kubeflow specifically.
- The identity lesson: every pod's service account is a credential, and when a front door is left open, its default permissions decide how far an attacker can go.
At a glance
| Organisations | Operators of tens of Kubernetes clusters running Kubeflow, seen by Microsoft's Azure Security Center; no victims named |
|---|---|
| When | Activity first seen in April 2020; disclosed by Microsoft on 10 June 2020 |
| Attacker | Unknown cryptomining operators; no attribution by Microsoft |
| Entry point | Kubeflow dashboards exposed to the internet after the istio-ingressgateway service was changed to a public load balancer |
| Identities abused | The Kubernetes service account mounted into Jupyter notebook servers, which by default can deploy containers in its namespace |
| Impact | Cluster compute, including machine learning nodes, used to run the XMRig cryptocurrency miner; no data theft reported |
| Category | NHI, LLM and AI platform. Incident class: confirmed NHI breach (mounted Kubernetes service account used to deploy cryptominers on machine learning clusters) |
What happened
Kubeflow is an open-source framework for running machine learning workloads on Kubernetes. It ships with a central dashboard and lets data scientists start Jupyter notebook servers inside the cluster. Microsoft's Azure Security Center, which monitors thousands of Azure Kubernetes Service clusters, had already reported a large campaign that abused exposed Kubernetes dashboards to deploy cryptocurrency miners. In April 2020 it saw something new: a suspect image from a public repository, ddsfdfsaadfs/dfsdf:99, being deployed on many different clusters. Its layers contained XMRig, a Monero miner, and the same repository held several related images with different mining configurations.
When Microsoft looked at the affected clusters, most of them ran Kubeflow. Yossi Weizman of Azure Security Center explained why such clusters appeal to miners: "Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs." By default, the Kubeflow dashboard is reached through the Istio ingress gateway, which is only accessible inside the cluster. "We believe that some users chose to do it for convenience," Weizman wrote of teams that changed the gateway service to a load balancer, which put the dashboard on the internet. "The attacker used an exposed dashboard (Kubeflow dashboard in this case) for gaining initial access to the cluster," Microsoft said. A tell-tale sign was the default namespace name "anonymous", which Kubeflow suggests on first use and which appeared widely in the campaign.
From the dashboard, Microsoft described two ways to run the miner. The first was to create a new Jupyter notebook server using the attacker's image. The second was to run code from an existing notebook server, which is itself a container with a mounted service account. "This service account (by default configuration) has permissions to deploy containers in its namespace," Microsoft wrote, and "The attacker managed to move laterally and deploy the container using the mounted service account." Microsoft said that while it had seen several campaigns against Kubernetes with a similar access vector, "this is the first time that we have identified an attack that targets Kubeflow environments specifically."
Microsoft did not name any affected organisations or describe stolen data. It published a kubectl query to find pods running the malicious image, a check on whether istio-ingressgateway has a public IP, and a list of mitigations. Security Affairs and SiliconANGLE reported the warning the following day. SiliconANGLE quoted Wei Lien Dang of StackRox: "Organizations should be mindful of the registries that users/clusters are allowed to download from".
Timeline
| Date | Event |
|---|---|
| April 2020 | Azure Security Center sees a suspect image containing XMRig deployed on many Kubernetes clusters, most running Kubeflow. |
| 10 June 2020 | Microsoft publishes "Misconfigured Kubeflow workloads are a security risk", describing the campaign and mitigations. |
| 11 June 2020 | Security Affairs and SiliconANGLE report Microsoft's findings. |
How it happened: the identity attack path
- Dashboard exposed. Some Kubeflow users changed the
istio-ingressgatewayservice to a load balancer, so the dashboard, internal by default, became reachable from the internet. - Clusters found. Attackers reached the exposed dashboards, often creating or using the default "anonymous" namespace.
- Code run in a notebook. Through the dashboard they could create a Jupyter notebook server with their own image, or run code inside a notebook server.
- Service account used. The notebook's mounted Kubernetes service account can deploy containers in its namespace by default; Microsoft says the attacker used it to move laterally and deploy the miner.
- Compute hijacked. Containers built from the attacker's image ran XMRig on the clusters' machine learning nodes.
Impact
- Confirmed: tens of Kubernetes clusters, most running Kubeflow, ran the attacker's cryptomining containers, according to Microsoft.
- Not reported: Microsoft did not report data theft, credential theft or named victims, and did not publish mining earnings or cost figures.
- Potential: a service account that can deploy containers, reached from an open dashboard, gives an attacker code execution inside the cluster. What else it could reach depends on the cluster's role-based access control and on the secrets visible in that namespace.
What this means for NHI and AI agent security
This belongs on an NHI list because the attacker did not need a stolen password. The identity that moved them from "can open a web page" to "can run containers" was the Kubernetes service account that every notebook pod receives automatically. Its token sits on disk inside the container, and its default rights in Kubeflow included creating workloads. Nobody issued that credential deliberately to the attacker; it came with the workload. That is the defining risk of workload identities: they are created by platforms, not people, and their permissions are set by defaults that few teams review.
The case is also an early example of AI infrastructure being targeted for its compute and identities. Machine learning platforms bundle dashboards, notebooks and powerful nodes, and they are often set up by data science teams for speed. Later campaigns against AI frameworks, such as ShadowRay against Ray clusters, followed a similar pattern of exposed interfaces and hijacked compute. Treat ML clusters as production: put authentication in front of every dashboard, and give notebook service accounts only the rights they need. Our Kubernetes NHI Security Guide and AI Infrastructure Workload Identity Guide cover these controls.
Recommendations
- Keep management dashboards off the internet. Leave the Kubeflow ingress gateway internal, check that
istio-ingressgatewayhas no public IP, and use policy to block public load balancers where they are not needed. See our Kubernetes NHI Security Guide. - Enforce authentication and access control on every ML front end. Dashboards and notebook servers should sit behind single sign-on, with per-user namespaces rather than a shared default.
- Cut service account permissions to the task. Notebook pods rarely need to create other workloads; remove that right, or disable automatic token mounting where the workload does not call the Kubernetes API. See the Service Account Security Guide.
- Allow only trusted images. Restrict which registries clusters can pull from and scan images before they run, as both Microsoft and StackRox advised.
- Monitor runtime behaviour of workload identities. Alert when a service account starts creating pods it has never created before, and on connections to mining pools. See the AI Infrastructure Workload Identity Guide.
- Inventory exposed endpoints continuously. Convenience changes such as a public load balancer are easy to forget; scan your own address space for dashboards and APIs. See the Cloud Workload Identity Guide.
Frequently asked questions
How were Kubeflow clusters hacked in 2020?
Attackers found Kubeflow dashboards that users had exposed to the internet by switching the Istio ingress gateway to a public load balancer. From the dashboard they started notebook servers or ran code in existing notebooks, and used the notebook's mounted Kubernetes service account to deploy containers running the XMRig cryptocurrency miner, according to Microsoft.
What role did the Kubernetes service account play?
Each Kubeflow notebook server runs with a mounted service account that, by default, can deploy containers in its namespace. Microsoft wrote that the attacker moved laterally and deployed the mining container using that service account, so a workload identity turned dashboard access into the ability to run new workloads.
Why do attackers target machine learning clusters?
Machine learning nodes are often powerful and sometimes include GPUs, which makes them valuable for cryptocurrency mining. They are also frequently set up quickly by data science teams, with dashboards and notebooks that may not get the same security review as production systems.
Related NHI Mgmt Group resources
TeamTNT Worm 2020 · Tesla Kubernetes Cryptojacking 2018 · ShadowRay 2024 · Kubernetes NHI Security Guide · Service Account Security Guide
How NHI Mgmt Group can help
Platform defaults create workload identities faster than teams can review them. We help security and platform teams find the service accounts their clusters create, cut their permissions to what each workload needs, and monitor how they are used. See our NHI and AI agent security training.
References
- Microsoft Security: Misconfigured Kubeflow workloads are a security risk (10 June 2020)
- Security Affairs: Microsoft discovers cryptomining campaign targeting Kubeflow tool for Kubernetes clusters (11 June 2020)
- SiliconANGLE: Microsoft warns of cryptomining attacks targeting Kubeflow on Kubernetes (11 June 2020)