GPU slicing and signed model artifacts change Kubernetes inference

At a glance

What this is: Jozu describes how fractional GPU scheduling on Kubernetes and signed model packaging can make inference workloads more efficient while reducing reliance on mutable public model hubs.

Why it matters: This matters because AI teams now have to govern both compute isolation and model provenance, which turns GPU utilisation, registry trust, and deployment integrity into one control problem.

By the numbers:

• A single A10 with 24GB of VRAM usually ends up dedicated to a workload that uses maybe 20% of the card.
• The demo ran Qwen3 4B on 60% of a single GPU on Kubernetes.
• The pod spec capped the workload at 60% of VRAM and 60% of compute cores.

Context

Inference workloads often waste GPU capacity because teams assign whole accelerators to models that need only a fraction of the hardware. That inefficiency becomes a governance issue when the same environment also depends on mutable model downloads at runtime, because the organisation is then managing both resource waste and unpinned supply-chain exposure at once.

Kubernetes gives teams a place to centralise scheduling, but it does not by itself solve GPU isolation or model provenance. This walkthrough shows the operational pattern: a fractional GPU slice for the workload, plus a signed and versioned model artifact delivered from a controlled registry rather than a public hub.

Key questions

Q: How should teams govern model artifacts in Kubernetes inference pipelines?

A: Treat the model artifact as a governed release object, not a download. Pin immutable digests, sign the package, store it in a controlled registry, and make the serving workload reference only that approved artifact. That gives you repeatable deployment, auditability, and rollback without trusting a mutable public hub at runtime.

Q: Why does runtime model download create supply chain risk for AI workloads?

A: Because the model that actually runs can differ from the one reviewed. A runtime fetch introduces a mutable external dependency, so availability, integrity, and version control all sit outside the deployment boundary. If the source changes, the workload can execute different weights without a new approval cycle.

Q: What should security teams do first when inference workloads waste GPU capacity?

A: Start by measuring actual utilisation per workload, then cap memory and compute per pod so the scheduler can pack multiple small jobs onto the same card safely. The goal is not just higher utilisation, but preventing one noisy workload from claiming the entire accelerator.

Q: How do organisations balance AI workload efficiency with supply chain integrity?

A: Use a single deployment boundary that covers both the compute slice and the artifact lineage. That means fractional GPU scheduling for utilisation, plus signed, immutable model packaging for provenance. Efficiency without provenance creates exposure, and provenance without isolation still wastes scarce hardware.

Technical breakdown

How GPU virtualization changes inference scheduling on Kubernetes

Project HAMi works by inserting a scheduler and device plugin that divide one physical GPU into fractional virtual GPUs. The pod asks for a memory percentage and a compute percentage, and the runtime enforces those limits inside the container. The key mechanism is interception at the CUDA layer, so the application sees a smaller GPU and cannot overclaim resources even if it tries. That is very different from informal time-slicing, which shares hardware but does not create the same hard boundary.

Practical implication: Treat fractional GPU allocation as a schedulable control, not just an efficiency tweak, and validate that the workload cannot exceed its assigned slice.

Why signed model artifacts matter more than runtime downloads

KitOps packages the model, configuration, tokenizer, code, and documentation into a ModelKit, which is an OCI-compliant artifact that can be signed and versioned. That matters because a model pulled at container start creates a moving target: the same deployment can fetch different bytes over time, and the runtime becomes dependent on an external hub being available. A signed artifact turns the model into something the organisation can approve, pin, and attest before execution.

Practical implication: Move model approval to build and registry time, then block deployments that depend on unsigned or mutable runtime fetches.

Why AI supply chain control now extends into the registry path

The article’s deeper point is that AI supply chain risk is not limited to weights alone. Once a model is shipped as a self-contained serving image, the registry, signature, digest, and access policy all become part of the trust boundary. That is similar to software supply-chain governance, but the payload is a model plus serving runtime rather than a binary alone. If the registry path is uncontrolled, the organisation still has an integrity gap even when the workload itself is isolated. Read the Ultimate Guide to NHIs: Standards.

Practical implication: Tie registry access, signature verification, and digest pinning to the same approval process that governs the model itself.

Threat narrative

Attacker objective: The attacker objective is to alter, substitute, or interrupt the AI model that the organisation thinks it is serving.

1. Entry occurs when the workload starts by pulling a model from an external hub or registry path that the organisation does not fully control.
2. Credential or artifact trust is then exposed if the model bytes, signature state, or version pinning can change between approval and execution.
3. Impact follows when the serving endpoint runs an untrusted or different model than the one the team intended, or when startup fails because the external source is unavailable.

Breaches seen in the wild

• TeamTNT worm 2020: A TeamTNT cryptomining worm spread through exposed Docker APIs and stole AWS credential files from every host it infected, Cado Security found.
• Massive Docker Hub Secrets Leak: 10,000+ Docker Hub container images expose hardcoded secrets and authentication keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI supply chain trust is now an identity problem, not just a packaging problem. Once a model ships as a signed artifact with its runtime, the registry path becomes part of the trust boundary that decides what code and weights are allowed to execute. That makes provenance, digest pinning, and approval workflow controls central to AI governance. Practitioners should treat model distribution as a controlled identity relationship between producer, registry, and runtime.

Fractional GPU scheduling changes the control model for AI infrastructure. Shared accelerators are no longer just a cost optimisation exercise when the scheduler can enforce hard memory and compute boundaries at the pod level. The practical governance question becomes whether the platform can prove a workload cannot overconsume what it was assigned. Teams that cannot answer that question are still running on allocation hope, not policy.

Containerised AI workloads now need the same boundary discipline that workload identity already demands. The article shows that a model, its weights, and its serving process can be packaged as one deployable unit, which reduces drift but also concentrates trust. Kubernetes NHI Security Guide becomes relevant because the same cluster controls that govern service accounts and workload identity also shape who can pull, schedule, and run the inference service.

Model immutability is becoming a governance control, not a convenience feature. A public hub that can change under a deployment creates approval drift, even if the rest of the platform is well controlled. Signed, content-addressed artifacts give teams a way to prove what was approved is what ran. That matters because AI operations are moving toward repeatable, attestable releases rather than ad hoc runtime fetches.

The strongest programmes will govern compute, artifact, and access paths together. Separating GPU efficiency, model provenance, and registry access into different workstreams leaves gaps that attackers or outages can exploit. The better pattern is a single release boundary that covers the model artifact, the serving image, and the cluster path used to start it. Practitioners should design AI deployment controls as one lifecycle, not three disconnected ones.

What this signals

Model provenance is becoming a deployment control, not a documentation exercise. The more inference teams rely on external model sources, the more they need approval workflows that lock both the bytes and the serving configuration before execution. The practical change is that release management now has to own model identity as well as container identity.

GPU efficiency and AI trust will converge in the same platform decisions. Teams will increasingly ask whether a scheduler can enforce hard resource boundaries while the registry enforces signed, immutable artifacts. That combined control plane is what turns ad hoc inference into something that can be audited and repeated.

For practitioners

• Enforce digest pinning for model artifacts Require every inference deployment to reference an immutable digest, not a mutable tag or branch, so the approved model bytes cannot change after review.
• Move model downloads behind your registry controls Block direct runtime pulls from public model hubs and serve models only from an internal registry with access control, signing verification, and audit logging.
• Set explicit GPU memory and compute caps Define per-workload limits for GPU memory and compute so a single inference pod cannot claim the full card when it only needs a fraction.
• Treat the model and server as one release unit Package the serving stack, weights, and configuration together so approval, rollback, and provenance checks apply to the same artifact.
• Verify cluster placement and runtime isolation Test that the scheduler, device plugin, and runtime report only the assigned slice to the container, then confirm the pod cannot see the rest of the GPU.

Key takeaways

• The article shows that inference efficiency problems and AI supply chain trust problems often appear in the same deployment path.
• Fractional GPU scheduling can improve utilisation, but only if the platform can enforce real isolation at the pod level.
• Signed, immutable model artifacts give teams a way to prove what was approved is what actually ran.

Related NHI Mgmt Group resources: Kubernetes NHI Security Guide · Ultimate Guide to NHIs: What are Non-Human Identities · Guide to SPIFFE and SPIRE

Key terms

• Fractional GPU: A fractional GPU is a way to share one physical GPU across multiple workloads instead of dedicating the whole device to a single service. It is useful when each workload needs only part of the GPU’s memory or compute capacity, but the exact isolation and scheduling behaviour depends on the sharing method used.
• ModelKit: An OCI-compliant package that bundles model weights, configuration, tokenizers, code, and supporting documentation into one versioned artifact. It gives AI teams a single object to sign, store, approve, and deploy instead of managing weights and runtime components separately.
• Rapid Inference Container: A prebuilt serving image that contains both the inference server and the model weights needed to expose an API. The control value is that the runtime becomes self-contained and reproducible, reducing startup-time dependency on external model downloads.
• Content-Addressed Artifact: An artifact whose identity is tied to its exact bytes rather than a mutable name or tag. This matters for AI release governance because it lets teams prove that the model they approved is the model that will actually run.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and workload identity. It helps practitioners connect identity control to the broader security decisions that shape modern deployment pipelines.