Because accelerator memory can hold shared inference state that is not governed like ordinary host memory. When multiple workloads or users share the same GPU pool, a weakness in isolation can expose KV cache fragments, prompt history, or other residual data across tenants.
How GPU multi-tenancy changes the trust boundary
GPU sharing changes the security model because the accelerator becomes a shared execution and memory resource, not just a compute device. In self-hosted deployments, that means isolation depends on more than host OS boundaries, because the GPU may retain state between tenants, multiplex work queues, and expose data through features intended for performance rather than tenancy separation.
A practical way to think about it is that the GPU becomes part of the confidentiality boundary for inference. If the platform does not enforce strong tenant separation, residual state can become reachable through cache, memory reuse, or scheduling interactions that are much harder to reason about than standard process memory on a single host.
Shared accelerators also make the blast radius of a misconfiguration larger. A single weak isolation decision can affect every workload mapped to the same card or pool, which is why multi-tenant GPU design belongs in the same conversation as access control, tenancy architecture, and data-handling policy.
What can leak across tenants on a shared GPU
The main concern is not that every tenant can directly read another tenant’s model weights. The concern is that inference-time artifacts and residual execution data may survive long enough to become observable to a neighbour. That includes KV cache fragments, prompt history, intermediate activations, scheduling residue, and other memory-backed state that was never intended to cross a workload boundary.
Risk increases when the operator assumes that container boundaries or application-level auth are enough. Those controls do not automatically govern accelerator-local state, so a design that looks separated at the API layer can still share sensitive artifacts at the hardware or driver layer.
That is why self-hosted deployments deserve special scrutiny when the same GPU pool serves different users, applications, or assurance levels. The more heterogeneous the tenants, the more valuable the shared resource becomes as a path for accidental exposure or deliberate probing.
Why self-hosted deployments are harder to isolate than they look
Self-hosting gives teams flexibility, but it also shifts responsibility for GPU partitioning, driver configuration, scheduling policy, and memory hygiene onto the operator. In practice, the security outcome depends on whether the stack actually supports hard tenancy boundaries, not whether the deployment is logically split in documentation.
This is especially important where operators rely on features such as GPU sharing, partitioning, or reuse for cost efficiency. Those features can be acceptable, but only when the environment is engineered and tested for leakage under the exact tenancy pattern being used. The same design that is safe for one trusted workload may be inappropriate for unrelated tenants with different confidentiality requirements.
For teams evaluating controls, this is a reason to treat GPU tenancy as a lifecycle and governance issue, not just an infrastructure tuning problem. Inventory, ownership, isolation assumptions, and data retention behaviour all matter because they determine whether sensitive inference state can persist beyond the session that created it.
Risk and Threat Considerations
Shared GPU memory and cache behaviour can create a cross-tenant confidentiality issue even when the host OS remains intact. The practical risk is that performance-oriented reuse, pooling, or imperfect teardown leaves recoverable state behind for the next workload on the same accelerator.
Failure mechanism: Isolation breaks when the deployment allows residual accelerator state, memory reuse, or insufficient partitioning between tenants, enabling one workload to observe artefacts from another.
Impact: Prompt content, conversation history, retrieval traces, or other inference-related data can be exposed across tenants, which can create data leakage, policy violations, and loss of trust in the shared platform.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-53 Rev 5, CSA Cloud Controls Matrix and NIST CSF 2.0 set the technical controls, while ISO/IEC 27001:2022 defines the regulatory obligations.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | IA-9 — Identification and Authentication (Non-Organizational Users) | Shared GPU tenants may rely on workload or service authentication at the platform boundary. |
| AC-6 — Least Privilege | GPU pooling risk is reduced by limiting which tenants can reach shared accelerator state. | |
| Recommendation — Apply IA-9 to ensure shared AI workloads are strongly authenticated before accessing GPU resources. Enforce AC-6 to restrict GPU access and reduce cross-tenant blast radius. | ||
| CSA Cloud Controls Matrix | IAM — Identity and Access Management | Cloud GPU tenancy depends on access controls, separation, and ownership over shared compute. |
| Recommendation — Use IAM controls to separate tenant access and manage who can share GPU capacity. | ||
| NIST CSF 2.0 | PR.AA-05 — Physical and Logical Access to Assets is Managed | GPU tenancy is an asset-access problem because shared accelerator resources must be bounded. |
| Recommendation — Manage logical access to shared GPU assets so tenant boundaries remain enforceable. | ||
| ISO/IEC 27001:2022 | A.8.5 — Secure authentication | Shared accelerator services need strong authentication around management and access paths. |
| Recommendation — Require secure authentication for access to GPU management and shared inference services. | ||
Practitioner Guidance
What to verify: Confirm that the GPU tenancy model matches the sensitivity of the workloads sharing it. If different users or business units share a pool, verify what state is actually cleared, what remains resident, and what the platform guarantees at reboot, job completion, and workload handoff.
Decision rule: If the platform cannot demonstrate hard separation for accelerator-local state, treat the GPU pool as shared confidential infrastructure and limit it to workloads with comparable trust and data sensitivity.
What practitioners underestimate: The dangerous assumption is that containerization or model-serving isolation automatically extends into the accelerator. In multi-tenant AI systems, the GPU can become the real trust boundary, so the question is not only whether the model is isolated, but whether the memory and execution path are too.
Practitioner takeaway: Multi-tenancy is risky when teams optimise for utilisation first and prove isolation second, because the GPU can preserve enough state to turn a performance shortcut into a cross-tenant exposure path.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 12, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org