Kubernetes inference on fractional GPUs: what changes for IAM teams?

TL;DR: A Qwen3 4B model can run on 60% of a single GPU in Kubernetes while the model weights and server ship as one signed, reproducible OCI artifact, according to Jozu, reducing waste and tightening AI supply chain control. The bigger shift is that inference efficiency and artifact trust now need to be governed together, not as separate problems.

Editorial analysis by NHI Mgmt Group, based on content published by Jozu: “Run Qwen3 4B on 60% of a GPU with HAMi and KitOps”.

By the numbers:

• A single A10 with 24GB of VRAM usually ends up dedicated to a workload that uses maybe 20% of the card.
• The demo ran Qwen3 4B on 60% of a single GPU on Kubernetes.
• The pod spec capped the workload at 60% of VRAM and 60% of compute cores.

Key questions

Q: How should teams govern model artifacts in Kubernetes inference pipelines?

A: Treat the model artifact as a governed release object, not a download.

Q: Why does runtime model download create supply chain risk for AI workloads?

A: Because the model that actually runs can differ from the one reviewed.

Q: What should security teams do first when inference workloads waste GPU capacity?

A: Start by measuring actual utilisation per workload, then cap memory and compute per pod so the scheduler can pack multiple small jobs onto the same card safely.

Practitioner guidance

• Enforce digest pinning for model artifacts Require every inference deployment to reference an immutable digest, not a mutable tag or branch, so the approved model bytes cannot change after review.
• Move model downloads behind your registry controls Block direct runtime pulls from public model hubs and serve models only from an internal registry with access control, signing verification, and audit logging.
• Set explicit GPU memory and compute caps Define per-workload limits for GPU memory and compute so a single inference pod cannot claim the full card when it only needs a fraction.

Bottom line: The article shows that inference efficiency problems and AI supply chain trust problems often appear in the same deployment path.

Explore further

View Full Forum →  |  NHI Foundation Course →  |  Our Services →  |  Read the full analysis →