GPU and AI Cluster Multi Tenancy Security: Isolating Shared Accelerators Between Tenants Safely

GPUs are now the most expensive shared resource in most clusters, and the controls that isolate them between tenants are weaker and less understood than the ones we use for CPU and memory. Platform teams slice accelerators with time-slicing, Multi-Instance GPU (MIG) and virtual GPUs to keep utilisation high, and GPU cloud providers (often called neoclouds) rent the same hardware to many customers. Each sharing mode gives a different isolation guarantee, and the host software stack that exposes GPUs to containers (driver, container toolkit, device plugin) runs with high privilege. This guide covers how to choose a sharing mode by trust level, harden the GPU software stack on Kubernetes, control who can reconfigure accelerators, keep tenant data from leaking through shared memory, and what to ask a GPU cloud provider. It is for platform, ML infrastructure and cloud security engineers. For Ray-specific controls see the Ray and AI Compute Cluster Security Guide, and for hardware-backed confidential computing see the Confidential AI and GPU Enclaves Guide.

Key takeaways

• Sharing mode is a trust decision. NVIDIA documents that time-slicing gives no memory or fault isolation between replicas, while MIG gives hardware memory protection and error isolation, so only MIG-class or whole-GPU allocation belongs between untrusted tenants.
• The GPU host stack is privileged code. Drivers, the container toolkit and the device plugin sit on the boundary between tenant and host, so patch them on a tracked cadence from the vendor's security bulletins.
• Reconfiguration rights equal control of the node. The MIG mig-config capability and CAP_SYS_ADMIN let a workload create, destroy and reach GPU instances; never grant them to tenant pods.
• Do not trust GPU memory to be clean. CERT/CC has documented GPU local memory leaking between processes on some hardware, so treat shared accelerators like shared caches and isolate sensitive models and data.
• Neocloud security is shared responsibility. Ask the provider in writing how GPUs are partitioned, wiped and patched between customers, and keep your own identity, network and secrets controls on top.

Why GPU multi tenancy is different

CPU and memory isolation has decades of kernel and hypervisor hardening behind it. GPU isolation is newer and depends on three layers that the tenant cannot see: the hardware partitioning features of the accelerator, the vendor kernel driver, and the user-space plumbing that hands a device to a container. A weakness in any of them can let one tenant observe another tenant's data, consume their capacity or reach the host.

Three properties make AI clusters a high-value target:

• Valuable contents: model weights, fine-tuning datasets, prompts and inference outputs sit in GPU memory in clear text unless confidential computing is in use.
• Privileged plumbing: the device plugin and container runtime hooks need host-level access to inject device nodes and driver libraries into containers.
• Costly compute: stolen GPU time is directly monetisable, which is why compute hijacking (ATT&CK T1496.001, Resource Hijacking: Compute Hijacking) is a common outcome when a cluster is exposed.

Choosing a GPU sharing mode

NVIDIA's documentation describes four ways to share an accelerator. They are not interchangeable from a security point of view.

Points that are easy to miss

• MIG compute instances are not separate tenants: a GPU instance can be split into compute instances, but NVIDIA states that compute instances inside one GPU instance share memory and engines. Give each tenant its own GPU instance, not just its own compute instance.
• Time-slicing hides who used what: NVIDIA notes that DCGM-Exporter cannot associate metrics with containers when time-slicing is enabled, which weakens chargeback and forensic attribution.
• Time-sliced requests are not proportional: asking for more than one time-sliced GPU does not guarantee a proportional share of compute, so noisy neighbours remain possible. Setting failRequestsGreaterThanOne: true stops tenants requesting several replicas.
• vGPU defaults to best effort: the best effort scheduler is the default on all supported architectures. Switch to equal share or fixed share where tenants need predictable performance.

Hardening the GPU software stack

On Kubernetes, the vendor driver runs on the node and a vendor device plugin advertises GPUs as a schedulable resource such as nvidia.com/gpu or amd.com/gpu (Kubernetes documentation, Schedule GPUs). A container runtime toolkit then injects device nodes, libraries and hooks into each GPU container. These components cross the tenant and host boundary, so a flaw in them can lead to container escape (ATT&CK T1611, Escape to Host).

1. Inventory the stack per node: record GPU model, driver version, container toolkit version, device plugin or GPU Operator version and firmware. Treat it as part of the hardware inventory (NIST CSF 2.0 ID.AM-01).
2. Subscribe to vendor bulletins: NVIDIA publishes security bulletins for GPU drivers and software at its product security page, offers email notification by product family, and states that from 1 October 2026 bulletins are also published on GitHub in Markdown, CSAF and CVE formats. Feed CSAF into your vulnerability tooling.
3. Patch on a defined SLA: roll driver and toolkit updates through a canary node pool first, then drain and update nodes in waves. Keep the previous driver image so you can roll back. This is flaw remediation under NIST SP 800-53 SI-2.
4. Prefer CDI device injection: the NVIDIA Container Toolkit supports the Container Device Interface (CDI), an open specification describing exactly which device nodes and files a container receives, and names devices such as nvidia.com/gpu=0 or nvidia.com/gpu=1:0 for a MIG device. Regenerate the CDI spec after driver changes and MIG reconfiguration, which the refresh service does not handle automatically.
5. Keep required hooks enabled: NVIDIA warns that disabling a required hook can expose more GPUs to EGL and Vulkan applications than the container requested. Disable only hooks that conflict.
6. Block privileged GPU pods: use admission policy to reject privileged: true, host PID and IPC, hostPath mounts of /dev and added capabilities such as SYS_ADMIN in tenant namespaces. See the Kubernetes Admission Control Guide.
7. Restrict tenant images: allow only signed images from approved registries, since tenant images bring their own CUDA user-space libraries.

Controlling who can reconfigure accelerators

MIG access is governed by NVIDIA capabilities exposed as device nodes under /dev/nvidia-caps. The MIG user guide states that the mig-config capability allows creating and destroying MIG instances on any MIG-capable GPU, that cgroups are the primary access control mechanism, and that a process with CAP_SYS_ADMIN implicitly has access to all NVIDIA capabilities. It also warns that granting a container mig-config implies access to every GPU or compute instance that could be created.

The service accounts that run GPU operators are non-human identities with node-wide reach. Inventory them, keep their RBAC narrow and alert on any new binding to them. See the Kubernetes NHI Security Guide and the Kubernetes RBAC Guide.

Scheduling and tenant placement

Placement is the cheapest isolation control because it keeps sensitive tenants off shared hardware entirely.

• Dedicated node pools: taint GPU pools per tenant class and require matching tolerations, so a tenant cannot land on another tenant's nodes by accident.
• Labels you trust: Kubernetes documents Node Feature Discovery for labelling GPU nodes and node affinity for targeting them. Make sure tenants cannot write node labels, or they can steer scheduling.
• Quotas on GPU resources: set a ResourceQuota on requests.nvidia.com/gpu (or the MIG resource names) per namespace so one tenant cannot exhaust the pool.
• Limits only: Kubernetes requires GPUs to be specified in limits; requests, if set, must equal limits. Validate this in admission to avoid scheduling surprises.
• Batch schedulers and job queues: whatever sits above Kubernetes (Slurm, Ray, queueing systems) must map each job to a tenant identity, not a shared service account. The Ray and AI Compute Cluster Security Guide covers Ray specifically.

Data leakage through shared GPU memory

CERT/CC vulnerability note VU#446598 (LeftoverLocals, published January 2024) describes GPU local memory that kept uninitialised data from earlier work, so a kernel from one process could read values left by another process or user on the same GPU. CERT/CC notes that deep learning operations rely heavily on local memory, so large language model workloads on shared GPUs may be exposed. Its guidance is to keep devices and libraries updated and apply vendor protections.

The lesson is broader than one issue: GPU memory should not be assumed clean between tenants.

• Separate by sensitivity: never co-locate a sensitive model or dataset with untrusted code on the same physical GPU in a time-sliced mode.
• Clean between jobs: for exclusive-GPU rentals, reset or reinitialise the GPU between tenants as part of the job teardown and record that it happened.
• Treat it as shared system resources: NIST SP 800-53 SC-4 (Information in Shared System Resources) and SC-39 (Process Isolation) are the right controls to map this risk to.
• Use confidential computing for the highest sensitivity: attested GPU enclaves protect data in use from the host operator; see the Confidential AI and GPU Enclaves Guide.

Working with GPU cloud providers

Neoclouds and hyperscaler GPU services take on part of the isolation stack. You cannot audit their hardware directly, so get answers in writing and keep your own controls in place.

On your side: use workload identity federation instead of static cloud keys on training nodes, keep model and dataset buckets private with narrow roles, and put inference and management endpoints behind authentication. The Tesla Kubernetes cryptojacking incident shows the pattern: an unauthenticated cluster console exposed cloud credentials stored in a pod, and attackers used the compute for mining.

Detection and measurement

• GPU telemetry per tenant: export utilisation, memory and error counters (for example with DCGM-Exporter) and attribute them to pods. Unexpected sustained utilisation outside job windows is a compute hijacking signal.
• Kubernetes audit events: alert on pods requesting privileged mode, host paths under /dev, SYS_ADMIN, or GPU resources in namespaces not entitled to them. See the Kubernetes Audit Log Detection Guide.
• Configuration drift: alert on changes to time-slicing and MIG ConfigMaps, device plugin DaemonSets and node labels.
• Driver and toolkit versions: report nodes running versions below the current bulletin baseline.

Common failure patterns

• Time-slicing used as a tenant boundary: enabling replicas to raise utilisation, then placing different teams or customers on the same GPU, even though the vendor states there is no memory or fault isolation.
• Compute instances mistaken for tenants: splitting one MIG GPU instance into compute instances for different tenants, which still share memory.
• Privileged GPU containers: granting privileged: true or SYS_ADMIN to "make the GPU work", which hands the tenant every NVIDIA capability and a path to the host.
• Stale driver stacks: GPU node images that are rebuilt rarely, so driver and toolkit fixes from vendor bulletins lag months behind.
• Exposed cluster interfaces with stored credentials: unauthenticated dashboards or job APIs on GPU clusters holding static cloud keys, as in the Tesla incident above.
• No tenant attribution: shared service accounts for job submission, so GPU usage and data access cannot be tied to a person or tenant.

Framework mapping

Practitioner checklist

• Classify every GPU tenant by trust level and map each class to an allowed sharing mode.
• Ban time-slicing between different teams or customers; use whole GPUs or separate MIG GPU instances.
• Inventory driver, container toolkit, device plugin and firmware versions on every GPU node.
• Subscribe to the GPU vendor's security bulletins and ingest CSAF advisories into vulnerability management.
• Patch the GPU stack through canary pools with a documented rollback image.
• Reject privileged mode, SYS_ADMIN and /dev host paths for tenant pods by admission policy.
• Keep mig-config and sharing ConfigMaps under platform-only control and alert on changes.
• Taint GPU pools per tenant class and set ResourceQuota on GPU resources per namespace.
• Reset GPUs between exclusive tenants and record it.
• Get written answers from any GPU cloud provider on partitioning, wiping, patch SLA and logs.

Standards and references

• NVIDIA: Multi-Instance GPU User Guide, Concepts
• NVIDIA: MIG Device Nodes and Capabilities
• NVIDIA: GPU Operator, Time-Slicing GPUs in Kubernetes
• NVIDIA: Container Toolkit, Support for Container Device Interface
• NVIDIA: Virtual GPU Software, Changing vGPU Scheduling Behaviour
• NVIDIA: Product Security Bulletins
• Kubernetes: Schedule GPUs
• CERT/CC: VU#446598 GPU local memory leakage

Related NHI Mgmt Group resources: Ray and AI Compute Cluster Security Guide · Confidential AI and GPU Enclaves Guide · Kubernetes Security Guide · Kubernetes NHI Security Guide