Anyone who can reach a Ray cluster's dashboard, Jobs API or Ray Client port can run arbitrary code on every node, and with it read every credential the cluster holds. Ray was built as a developer tool for trusted networks, so keeping it off untrusted networks and turning on its token authentication are customer-side controls, especially on GPU clusters full of cloud keys, model hub tokens and AI API keys. This guide covers how access works in Ray and KubeRay, the token authentication Ray now ships, the credentials and service accounts that make a compromised cluster expensive, what to log, and a hardening baseline. It is written for platform engineers, ML infrastructure teams and security engineers who run or review Ray, and applies by analogy to other distributed AI compute schedulers.
This is an independent guide from NHI Mgmt Group, based on the Ray project's public documentation as of November 2026. It is not sponsored or reviewed by Anyscale. Product features change, so confirm settings against the Ray project's current documentation.
Key takeaways
- Reachability equals code execution. Ray's own security documentation states that any client that can reach the dashboard, Jobs or Client ports can run arbitrary code, explicitly through jobs, indirectly through dashboard REST APIs, or implicitly through cloudpickle deserialisation.
- The network is the primary control. Ray expects isolation to be enforced outside the cluster: private networks, security groups, NetworkPolicies and authenticating proxies come first.
- Token authentication is defence in depth, not a perimeter. Available from Ray 2.52.0 and on by default for all clusters from Ray 2.61, the token is a shared, non-expiring secret sent as an HTTP header, so it needs TLS and careful storage.
- Ray has no internal isolation between users or jobs. Separate teams, tenants and trust levels need separate clusters, each with its own narrowly scoped workload identity.
- The blast radius is the credentials on the nodes. Nodes can hold cloud keys, AI API, model hub and SaaS tokens, SSH keys and Kubernetes secrets. Short-lived workload identity and scoped API keys limit what an attacker can take.
How identity and access work in Ray
A Ray cluster has a head node and worker nodes. Core components communicate over gRPC for distributed memory, heartbeats, metrics and logs. Three additional services sit on top, and Ray's documentation is explicit that each gives complete access to the cluster and its compute:
- Ray dashboard: cluster introspection and debugging, served over HTTP on port 8265 by default. Its REST APIs can trigger code execution.
- Ray Jobs: runs alongside the dashboard and accepts job submissions (the
/api/jobs/endpoints). A job is an entrypoint command plus a runtime environment, so a submitted job is a shell command on the cluster. - Ray Client: interactive development against a remote cluster (port 10001 by default). Objects cross the wire with cloudpickle, so Python's pickle trust model applies: deserialising untrusted data is code execution.
Turn on token authentication: without it, these services accept any caller that reaches them. Users of one cluster share the same access and jobs run side by side, so use separate clusters where teams or jobs need isolation. In identity terms, every caller who reaches a port is effectively root on the cluster and inherits the cluster's workload identity: the cloud instance role, the Kubernetes service account and any secrets in environment variables or mounted files.
Why scanners are not enough
Anyscale, Ray's maintainer, states that Ray must only be reachable from trusted machines. Many scanners and software composition tools do not report an exposed Ray cluster, so do not rely on vulnerability scanning to find one; check exposure directly, and apply Ray updates listed on the Ray security page promptly.
Token authentication
| Setting or behaviour | What it does | Security note |
|---|---|---|
RAY_AUTH_MODE=token | Enables token authentication on nodes and clients | Available from Ray 2.52.0; default for local clusters in 2.59 and all clusters in 2.61 |
RAY_AUTH_MODE=disabled | Opts out of authentication | Treat as a policy exception; alert if seen in cluster configs or images |
RAY_AUTH_TOKEN | Supplies the token value directly | Discouraged in the docs: any code in the process environment can read it |
RAY_AUTH_TOKEN_PATH | Points to a token file | Preferred; mount from a secret store with restrictive file permissions |
~/.ray/auth_token | Default token location, plaintext | Protect on shared machines; never commit to version control |
ray get-auth-token --generate | Prints or creates the token | Generation is a privileged action: run it in the deployment pipeline, not on laptops |
| Dashboard sign-in | Token entered once and stored as a cookie for up to 30 days | Long-lived browser credential; shared or kiosk browsers must not be used |
| Transport | Token sent as an HTTP header | Over plain HTTP it can be intercepted; use a TLS proxy, SSH tunnel, kubectl port-forward or VPN |
| Lifetime and rotation | Tokens do not expire; one token per cluster lifetime | Rotation means restarting the cluster with a new token and updating every client |
The token check is the lookup order RAY_AUTH_TOKEN, then the file at RAY_AUTH_TOKEN_PATH, then ~/.ray/auth_token. A stale environment variable therefore silently overrides a rotated token file, which is a common cause of confusing failures after rotation. Note also what token mode is not: it is a single shared secret, so it proves "holds the cluster token", not which person or service submitted a job. Attribution still needs an authenticating proxy or the Kubernetes RBAC option that KubeRay documents.
Admin roles and privileged access
Ray has no internal role model, so "admin" is defined by who controls the layers around it. Map these explicitly:
| Layer | Privileged action | Who should hold it |
|---|---|---|
Cluster launcher (ray up) or KubeRay RayCluster / RayJob resources | Create clusters, choose images, mount secrets, set the service account | Platform team through CI/CD; developers through a reviewed template |
| KubeRay operator | Reconciles Ray resources cluster-wide | Operator service account scoped to its namespaces, not cluster-admin |
| Ray auth token | Full code execution on that cluster | Only the users and pipelines of that cluster; stored in a secrets manager |
| Cloud IAM role on nodes | Whatever the role allows: storage, model registries, other services | Scoped per cluster to the buckets and APIs the workload needs |
| Network path to 8265 and 10001 | Reach the dashboard, Jobs and Client | Identity-aware proxy or bastion; never a public load balancer |
- Do not run Ray as root: running as root turns code execution into host compromise. Use a non-root container user and drop Linux capabilities.
- Cluster-admin is a red flag: an operator with administrator rights widens any compromise. Give the Ray pods' service account no Kubernetes API permissions unless autoscaling or a specific integration needs them. See the Kubernetes RBAC Guide.
- Human access through an identity-aware proxy: put dashboard access behind SSO with MFA so each session is attributable. See the Identity-Aware Proxy Guide.
Integrations, API tokens and service accounts
A Ray cluster is a concentration point for non-human identities. Training and batch inference jobs pull data from object storage, models from hubs, and call external AI APIs, so the nodes collect credentials. Plan against the full inventory below.
| Credential on the cluster | Typical location | Better pattern |
|---|---|---|
| Cloud access keys (AWS, Google Cloud, Azure, GPU clouds) | Environment variables, ~/.aws/credentials, job runtime_env | Instance role or Kubernetes workload identity with short-lived tokens, scoped per cluster |
| Instance metadata credentials | IMDS endpoint reachable from every pod | IMDSv2 with hop limit, block metadata from worker pods that do not need it |
| Hugging Face tokens | Environment, ~/.cache/huggingface | Read-only, fine-grained tokens; write tokens only in a separate publishing pipeline |
| OpenAI and other AI API keys | Environment variables in job specs | Project-scoped keys with spend limits, injected per job from a secrets manager |
| Database and SaaS tokens (Stripe, Slack) | Config files, job arguments | Keep out of compute clusters; call through a service that holds the credential |
| SSH keys | Head node home directories (cluster launcher) | Short-lived SSH certificates or no SSH; use port-forwarding through the control plane |
| Kubernetes service account token | Mounted in every pod by default | Disable automounting where not needed; bound, audience-scoped tokens elsewhere |
- Model hub write access: stolen Hugging Face write tokens could let an attacker replace an organisation's models with malicious versions. A cluster that only needs to download models should never hold a write token.
For the wider pattern of replacing static keys on GPU and training infrastructure with federated workload identity, see the AI Infrastructure Workload Identity Guide and the Kubernetes Service Account Token Guide.
Logging and detection
Detection for Ray depends on runtime telemetry and on the identity systems around the cluster. The dashboard's own job history records submitted commands, so review it.
| Signal | Source | What suspicious looks like |
|---|---|---|
| Job submissions | Ray Jobs history, dashboard logs, proxy access logs | Submissions from unknown source IPs, entrypoints with base64 strings, curl or wget piped to a shell, reads of /etc/shadow |
Requests to /api/jobs/ without a proxy identity | Load balancer or proxy logs | Any direct request bypassing the authenticating proxy |
| Outbound connections from nodes | VPC flow logs, DNS logs, egress proxy | Mining pools, tunnelling services, callback domains used for exploit confirmation, new external IPs |
| GPU and CPU utilisation | Cloud monitoring, DCGM or node exporter | Sustained full utilisation with no scheduled job, processes with masqueraded names |
| Persistence | Host and container runtime telemetry | New cron entries, unexpected processes on every node at once (a sign of node affinity scheduling abuse) |
| Credential use | CloudTrail, Google Cloud audit logs, Azure activity log, AI provider usage dashboards | Node role credentials used from outside the cluster's network, AI API spend spikes, Hugging Face token use from new locations |
| Kubernetes API activity | Kubernetes audit log | Ray pod service accounts listing secrets or creating pods outside the operator's pattern |
Ingest Ray job history and proxy logs into the SIEM rather than leaving them on the head node, where an attacker with code execution can delete them. See the Kubernetes Audit Log Detection Guide.
Metrics that show the controls are working
- Externally reachable Ray ports: count of clusters with 8265, 10001 or the GCS port reachable from outside the trusted network. Target zero, measured by an external scan.
- Clusters with authentication enabled: percentage of clusters running with
RAY_AUTH_MODE=tokenand behind an authenticating proxy. Target 100 per cent. - Static credentials on nodes: number of long-lived cloud keys or AI API keys found in cluster images, environment variables and job specs. Trend to zero.
- Attributable job submissions: percentage of jobs whose submitter maps to a named person or pipeline identity.
- Token age: days since each cluster's token was rotated, with a ceiling set by policy and a forced rotation on staff departure.
Hardening baseline
Roll the baseline out in this order. Each step reduces exposure on its own, so do not wait for the full set.
- Find every cluster. Inventory Ray clusters across cloud accounts and Kubernetes namespaces (KubeRay
RayCluster,RayJobandRayServiceresources, plus VMs started withray start). Scan your external ranges for 8265 and 10001. Anyscale publishes a Ray open ports checker for this purpose. - Remove internet exposure. Bind the dashboard to a private interface rather than
0.0.0.0, delete public load balancers and Ingress objects for the dashboard, and restrict security groups to the head node's private network. Copied examples and boilerplate can expose the dashboard on all interfaces: review copied templates. - Treat exposed clusters as compromised. If a cluster was reachable without authentication, rotate every credential it held, review job history and cloud audit logs, and rebuild nodes from clean images. Use the Cryptojacking Incident Response Playbook for the response.
- Put humans behind an authenticating proxy. Access to the dashboard through an identity-aware proxy with SSO and MFA, or through
ray dashboardSSH forwarding orkubectl port-forwardfor administrators. - Enable token authentication. Upgrade to a Ray version that supports it, set
RAY_AUTH_MODE=token, distribute the token withRAY_AUTH_TOKEN_PATHfrom a secret, and test job submission and Ray Client from each pipeline before enforcing. Roll back by reverting the cluster template, not by settingdisabledin production. - Encrypt in transit. Terminate TLS at the proxy for dashboard and Jobs traffic and enable TLS for gRPC where the KubeRay TLS guidance fits your environment. Ray's documentation is clear that TLS does not replace network isolation.
- Isolate with NetworkPolicies and egress control. Allow only intra-cluster traffic between Ray pods, ingress to 8265 and 10001 from the proxy only, and egress to named destinations (object storage, model hub, AI APIs) through an egress proxy. See the Agent Egress Control and Network Isolation Guide.
- Replace static secrets. Move to workload identity for cloud access, block or harden instance metadata, scope Hugging Face and AI API tokens to read-only and per project, and disable service account token automounting where unused.
- Harden the runtime. Non-root user, read-only root filesystem where practical, no privileged containers, signed and pinned images, and a dependency policy for runtime_env packages.
- Separate clusters by tenant and trust. One cluster per team or sensitivity level, ephemeral clusters for untrusted experiments, each with its own token and role. For running model-generated or user-supplied code, see the AI Agent Sandboxing and Code Execution Guide.
Test each change in a non-production cluster first: submit a known job through every path (CLI, SDK, Ray Client, any orchestration tool such as Airflow or a notebook), confirm an unauthenticated request is refused, and only then apply the template to production clusters.
Choosing an access pattern
| Pattern | Fits | Trade-off |
|---|---|---|
Private network plus SSH or kubectl port-forward | Small teams, a handful of clusters | Simple and strong; attribution relies on SSH or Kubernetes identity, awkward for many users |
| Identity-aware proxy plus token auth | Shared platforms with many developers | Per-user attribution and MFA; the proxy must hold or inject the cluster token, so it becomes a privileged component |
| Kubernetes RBAC authentication (KubeRay) | Organisations already governing access through Kubernetes | Reuses existing RBAC and audit; depends on mature Kubernetes identity governance |
| Managed Ray service | Teams that do not want to operate the control plane | Provider handles dashboard exposure and auth; you still own node credentials and network egress |
Small organisations get most of the benefit from the first row plus token authentication. Large platforms with many tenants need the second or third row, because a shared token alone cannot tell you who ran a job.
Common misconfigurations
- Dashboard on a public IP: anyone who reaches it can submit jobs. Keep it on a private network behind authentication.
- Relying on the scanner: a clean vulnerability report says nothing about exposure. Check reachability, not CVE lists.
- Cloud keys in environment variables: the first thing a submitted job can print.
- One shared cluster for everyone: Ray does not isolate jobs, so a single compromised notebook or dependency exposes every team's data and credentials.
- Ray pods with cluster-admin or broad Kubernetes permissions: turns a Ray compromise into a Kubernetes compromise. See the Kubernetes Cluster Compromise Playbook.
- Unrestricted egress: lets miners reach pools, reverse shells reach their operators and stolen data leave. Default-deny egress makes most post-exploitation steps fail.
Framework mapping
| Risk in Ray | Threat reference | Controls |
|---|---|---|
| Unauthenticated Jobs API on a reachable dashboard | ATT&CK T1190 Exploit Public-Facing Application; ATLAS AML.T0049 | NIST SP 800-53 SC-7, AC-3, CM-7; CSF 2.0 PR.IR-01; ISO/IEC 27001:2022 A.8.20, A.8.22; CIS Controls v8.1 Control 12 |
| Arbitrary Python execution through jobs or Ray Client | ATT&CK T1059.006 Python | NIST SP 800-53 AC-6, CM-7; ISO/IEC 27001:2022 A.8.5 |
| Secrets in files, environment and metadata | ATT&CK T1552.001, T1552.005; ATLAS AML.T0055 Unsecured Credentials | NIST SP 800-53 IA-5; CSF 2.0 PR.AA-05; CIS Controls v8.1 Control 3 and 6 |
| Stolen cloud credentials used elsewhere | ATT&CK T1078.004 Cloud Accounts | NIST SP 800-53 AC-6, SI-4; CSF 2.0 DE.CM-01 |
| GPU cryptomining and AI spend abuse | ATT&CK T1496.001 Compute Hijacking; ATLAS AML.T0034 Cost Harvesting | NIST SP 800-53 SI-4, AU-2; ISO/IEC 27001:2022 A.8.16; CIS Controls v8.1 Control 13 |
| Token sent in clear over HTTP | Interception on the network path | NIST SP 800-53 SC-8 |
Practitioner checklist
- Inventory every Ray cluster, including KubeRay resources and VM clusters, and scan external ranges for ports 8265 and 10001.
- Remove all public exposure of the dashboard, Jobs API and Ray Client; bind to private interfaces only.
- Treat any cluster that was reachable without authentication as compromised: rotate its credentials and review job history.
- Enable
RAY_AUTH_MODE=token, distribute the token by file from a secrets manager, and give each cluster its own token. - Put human access behind an identity-aware proxy with SSO and MFA, terminating TLS.
- Replace static cloud keys with workload identity, harden instance metadata, and scope model hub and AI API tokens to read-only and per project.
- Run Ray as non-root, remove Kubernetes API permissions from Ray pods, and apply NetworkPolicies with default-deny egress.
- Use separate clusters for separate teams and trust levels; never mix untrusted experiments with production credentials.
- Ship job history, proxy logs, flow logs and cloud audit logs to the SIEM and alert on unattributed job submissions and GPU use without scheduled jobs.
- Review cluster templates and Helm values for
0.0.0.0bindings andRAY_AUTH_MODE=disabledbefore they are reused.
Standards and references
- Ray Project: Security
- Ray Project: Token Authentication
- Ray Project: Configure Ray clusters to use token authentication (KubeRay)
- Ray project: Security
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: AI Infrastructure Workload Identity Guide · MLflow and Kubeflow Security Guide · vLLM, Triton and NVIDIA NIM Inference Server Security Guide · Kubernetes NHI Security Guide · LLMjacking Guide