Production inference servers such as vLLM, NVIDIA Triton Inference Server and NVIDIA NIM are built to sit inside a trusted network behind a gateway, yet they are often deployed with their full API surface reachable, including endpoints that pause generation, load adapters, change model weights or load code from a model repository. Their own documentation is explicit: vLLM states that its API key protects specific path prefixes and that inter-node traffic should be secured by the operator on a trusted network, and Triton states that model and backend code should come only from trusted sources. This guide covers how access control works in each server, the privileged operations that need protecting, the credentials these servers hold, logging, a hardening baseline and the misconfigurations that recur. It is written for ML platform, infrastructure and security teams running GPU serving at scale. Workstation tools such as Ollama and LM Studio, and the general self-hosting pattern, are covered in the Self-Hosted LLM Security Guide; this page goes deeper on production serving stacks.
Key takeaways
- None of these servers is an authentication boundary. vLLM's
--api-keycovers only/v1,/v2,/inferenceand/cohere; Triton expects a gateway to handle authorisation. Put an identity-aware gateway in front and allowlist the endpoints users need. - Control endpoints are the real risk. vLLM exposes
/pause,/update_weightsand/abort_requestswithout a token, and Triton's model control APIs can load models whose backends execute code. Block them at the proxy. - The model repository is code. Triton warns that dynamic model repository updates can lead to arbitrary code execution, and that write access to backend, repository agent and cache directories must be controlled.
- Distributed serving trusts the network. PyTorch Distributed, KV cache transfer and Triton or vLLM gRPC interfaces have no authentication by default; isolate multi-node traffic on a dedicated network.
- Serving hosts carry high-value credentials. Hugging Face tokens, NGC API keys, cloud storage roles and registry credentials live on these nodes, and vLLM propagates the driver's environment to Ray workers by default.
How identity and access work in vLLM, Triton and NIM
| Aspect | vLLM OpenAI-compatible server | NVIDIA Triton Inference Server | NVIDIA NIM |
|---|---|---|---|
| Built-in caller authentication | Bearer token from --api-key or VLLM_API_KEY, for /v1, /v2, /inference and /cohere paths only | None for inference by design; shared-secret header restriction for chosen API categories with --http-restricted-api and --grpc-restricted-protocol (beta) | Containerised microservice; treat the served endpoint as unauthenticated and front it with a gateway (check NVIDIA's current NIM documentation) |
| Unauthenticated surface | /invocations, /pooling, /classify, /score, /rerank, /pause, /resume, /update_weights, /scale_elastic_ep, /metrics, /tokenize, /health and others | Health, metadata, inference, shared memory, model repository, statistics, tracing and logging APIs enabled in the pre-built server | As exposed by the container's API |
| Transport security | Terminate TLS at the proxy; gRPC (--grpc-port) has no authentication, authorisation or encryption | --grpc-use-ssl and --grpc-use-ssl-mutual, both default False | Terminate TLS at the gateway or ingress |
| Intended placement | Behind a reverse proxy that allowlists endpoints | Microservice inside a service mesh or behind an API gateway, never directly on an untrusted network | Inside a cluster or private network behind an ingress |
vLLM
- API key scope: the vLLM security documentation warns not to rely on
--api-keyalone. The SageMaker-compatible/invocationsendpoint runs the same inference as/v1without a token, so a client can bypass the key entirely. - Development and profiler endpoints:
VLLM_SERVER_DEV_MODE=1adds/collective_rpc, which the documentation calls extremely dangerous because it executes arbitrary RPC methods on the engine, plus cache reset and sleep endpoints. Never set it in production, and do not enable profiler endpoints there. - Endpoint plugins: routes added through the
vllm.endpoint_pluginsgroup are unauthenticated unless they sit under a protected prefix. They load only when named inVLLM_PLUGINS; review every named plugin's routes before deployment.
Triton
- Gateway first: NVIDIA's secure deployment guide describes Triton handling only trusted, validated requests, with ingress controllers or a service mesh doing authorisation, encryption and rate limiting.
- Restricted APIs:
--http-restricted-apiand--grpc-restricted-protocolrequire a specific header key and value for chosen API categories, which can limit model control APIs to trusted operators. A static shared secret is a weak control on its own; use it as a second layer behind the gateway. - Protocol switches:
--allow-http,--allow-grpcand--allow-metricsdefault to True,--allow-sagemakerto False, and--allow-vertex-aidepends onAIP_MODE. Turn off every protocol you do not use.
Admin roles and privileged access
These servers have no admin users; privilege lives in endpoints, flags, files and the orchestration layer.
| Privileged capability | Server | Why it matters | Control |
|---|---|---|---|
| Model control APIs and repository polling | Triton (--model-control-mode explicit or poll) | Loading a model can run backend code from the repository; NVIDIA warns this can lead to arbitrary code execution | Keep the default none in production; if dynamic loading is needed, restrict the APIs and repository write access to the deployment pipeline |
| Backend, repository agent and cache directories | Triton (/opt/tritonserver/backends, repoagents, caches) | Adding untrusted shared libraries leads to arbitrary code execution | Read-only container layers; no write mounts |
| Weight updates and LoRA loading | vLLM (/update_weights; /v1/load_lora_adapter with --enable-lora and VLLM_ALLOW_RUNTIME_LORA_UPDATING=True) | Changes model behaviour at runtime; vLLM states dynamic LoRA loading is not a secure operation | Block at the proxy; leave runtime LoRA updating off for untrusted clients |
| Pause, abort and scaling controls | vLLM | Unauthenticated denial of service | Block at the proxy; expose only to the orchestrator |
| Tool servers and code interpreter | vLLM (--tool-server demo) | Model-generated code runs in a Docker container that, by default, inherits host Docker networking and may reach cloud metadata | Leave off unless needed; omit code_interpreter from VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS; use a stricter custom sandbox |
| Cluster and node access | All | Anyone who can exec into a serving pod can read tokens and models | Kubernetes RBAC, no shell access in production, privileged access through just-in-time elevation. See the Kubernetes RBAC Guide |
Triton's pre-built containers include a non-root user, triton-server with user ID 1000, which NVIDIA recommends using via the container's user option. Run vLLM and NIM containers as non-root as well, with dropped capabilities and read-only root filesystems where the image supports it.
Integrations, API tokens and service accounts
- Model hub tokens:
HF_TOKENfor gated Hugging Face downloads and the NGC API key that authorises NIM container pulls and model downloads from NVIDIA's NGC registry are long-lived credentials. Scope them to read-only, use organisation-owned accounts, and prefer pulling models into an internal registry during build so serving nodes need no hub token at runtime. See the Model Supply Chain Guide. - Ray worker environment propagation: vLLM's documentation states that, with Ray, every driver environment variable is copied to workers except a small denylist, so
HF_TOKEN, cloud keys and registry tokens set on the API server reach worker processes, where another process under the same OS user may read them. Add names toray_non_carry_over_env_vars.jsonunder$VLLM_CONFIG_ROOT, and inject credentials through a secrets manager or mounted file instead of the driver's environment. See the Ray and AI Compute Cluster Security Guide. - Model repository and artefact storage: Triton can load models from cloud object storage. The identity that reads the repository should be read-only, scoped to one prefix, and separate from any identity that can write to it.
- vLLM API keys: a single static key shared by all clients gives no attribution. Hold the key in the gateway, give each consuming application its own identity at the gateway, and rotate the backend key on a schedule. See the AI Gateway and LLM Gateway Guide.
- Kubernetes service accounts: NVIDIA recommends running Triton pods with the fewest possible permissions. Serving pods rarely need Kubernetes API access; set
automountServiceAccountToken: falseunless they do. See the Kubernetes Service Account Token Guide. - Cloud instance roles: GPU nodes often run with broad roles for storage and registries. The xinference PyPI compromise put a credential stealer inside an inference server package that collected AWS role credentials from instance metadata. Block metadata access from serving containers and scope node roles tightly.
Logging and detection
| Source | What to collect | Detections worth building |
|---|---|---|
| Gateway or reverse proxy | Caller identity, path, status, request size, tokens if the gateway counts them | Any request to control paths (/pause, /update_weights, model control APIs); /invocations or other unlisted paths reaching the backend; volume anomalies per caller |
Server metrics (/metrics, Triton Prometheus metrics) | Throughput, queue depth, KV cache use, model load events | Unexpected model loads or unloads; sustained load outside business patterns. Keep metrics endpoints internal: Triton recommends disabling them if not needed to avoid exposing sensitive information |
| Kubernetes audit and runtime security | Pod exec, config changes, new containers, process starts | Shells or package managers in serving containers; writes to model or backend directories; changes to server flags such as dev mode |
| Network flow logs | Connections between nodes and outbound traffic | Connections to PyTorch Distributed, KV transfer or gRPC ports from outside the serving network; egress to unknown hosts; metadata endpoint access |
| Model registry and storage logs | Writes to model repositories and adapter stores | Model or adapter changes outside the deployment pipeline |
For detection mapping, use Exploit Public-Facing Application (T1190), Cloud Instance Metadata API (T1552.005) and Compute Hijacking (T1496.001) in ATT&CK, and AI Model Inference API Access (AML.T0040), Manipulate AI Model (AML.T0018) and Cost Harvesting (AML.T0034) in MITRE ATLAS.
Metrics that show the controls work:
- Percentage of inference endpoints reachable only through an authenticating gateway (target 100%).
- Number of control or development endpoints reachable from client networks (target zero).
- Days from a vendor security bulletin to patched production for each serving stack.
- Number of serving nodes holding long-lived hub, registry or cloud keys at runtime (target zero).
- Percentage of production models loaded from a signed or hash-verified internal registry.
Hardening baseline
- Front every server with an identity-aware gateway: allowlist only the endpoints users need (for example
/v1/chat/completionsand/v1/modelsfor vLLM, or the specific Triton inference routes), authenticate callers with your identity provider or mTLS, rate limit, and log. Block everything else, including/invocations, metrics and control paths. - Bind and segment: bind servers to private interfaces, set
VLLM_HOST_IPexplicitly, and place multi-node traffic (PyTorch Distributed, KV cache transfer on--kv-portdefault 14579, data parallel master port default 29500) on an isolated network. vLLM notes that PyTorch'sTCPStorelistens on all interfaces by default, even on a single host, so firewall everything except the API port. - Freeze the model surface: run Triton with
--model-control-mode none,--disable-auto-complete-config,--exit-on-error trueand--strict-readiness true; leave vLLM runtime LoRA updating, dev mode, profiler, tokenizer info and tool servers off. - Turn off unused protocols: Triton
--allow-grpc falseor--allow-http falseas appropriate,--allow-sagemaker false,--allow-vertex-ai false,--allow-metrics falsewhere metrics are scraped another way, and tracing off unless needed. If gRPC must cross a network boundary, set--grpc-use-ssl-mutual true. - Bound resource use: set
VLLM_MAX_N_SEQUENCESto a workload-appropriate value (vLLM suggests 64 or 128 for public-facing deployments, against a default of 16384), keep media size limits enabled, and set--allowed-media-domainswithVLLM_MEDIA_URL_ALLOW_REDIRECTS=0to prevent SSRF through multimodal inputs. For Triton decoupled gRPC streaming to untrusted clients, set a bounded--grpc-max-response-pool-size. - Lock down files and identity: read-only model, backend, repository agent and cache directories; non-root users; no service account token unless required; metadata access blocked; credentials injected from a secrets manager and excluded from Ray propagation.
- Patch on bulletins: subscribe to NVIDIA product security bulletins and vLLM security advisories. Apply each fixed release promptly; NHI Mgmt Group recommends tracking the server version of every deployment so you can find and update all instances quickly.
Roll out in stages: deploy the gateway in front of a staging server with logging only, compare its endpoint log against the allowlist for a week, then enforce. Apply server flag changes one serving pool at a time behind the load balancer so you can drain and roll back a pool if latency or readiness regress. Small teams with one model can use a single reverse proxy and fixed flags; large platforms should standardise a hardened serving image and admission policies that reject dev mode, writable model mounts and missing gateway sidecars. See the Kubernetes Admission Control Guide.
Common misconfigurations
- Trusting
--api-key: a vLLM server published with only the API key, leaving/invocationsand control endpoints open to anyone who can reach the port. - Explicit model control for convenience: Triton run with
--model-control-mode explicitand its HTTP port reachable from client networks, so anyone who can reach it can load and unload models. - Flat GPU networks: multi-node serving where PyTorch Distributed and KV transfer ports are reachable from the wider cluster or VPC.
- Dev mode in production:
VLLM_SERVER_DEV_MODE=1left in a deployment manifest after debugging, exposing/collective_rpc. - Tokens in the driver environment:
HF_TOKENand cloud keys exported on the vLLM API server and silently copied to every Ray worker. - Unauthenticated AI compute in general: the ShadowRay campaign showed attackers using exposed, unauthenticated AI infrastructure to steal cloud credentials, AI API tokens and compute. Inference servers with open control APIs present the same opportunity.
Framework mapping
| Control area | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | OWASP |
|---|---|---|---|
| Gateway authentication and endpoint allowlisting | AC-3, IA-9, SC-7 | PR.AA-05 | API2:2023 Broken Authentication |
| Disabling control, dev and unused endpoints | CM-7, CM-6 | PR.PS-04 | API9:2023 Improper Inventory Management |
| Network isolation of distributed traffic | SC-7, SC-8 | DE.CM-09 | LLM10 Unbounded Consumption |
| Model repository integrity | SR-3, SR-4, CM-5 | GV.SC-06 | LLM03 Supply Chain; LLM04 Data and Model Poisoning |
| Credential handling on serving nodes | IA-5, AC-6, SC-28 | PR.AA-05 | LLM02 Sensitive Information Disclosure |
| Patching and monitoring | SI-2, RA-5, SI-4, AU-12 | DE.CM-09 | LLM10 Unbounded Consumption |
Practitioner checklist
- Inventory every vLLM, Triton and NIM deployment with version, owner, exposure and models served.
- Put an authenticating gateway with an endpoint allowlist in front of each server.
- Block
/invocations, control, metrics, development and profiler endpoints from client networks. - Run Triton with
--model-control-mode noneand read-only model, backend and agent directories. - Isolate PyTorch Distributed, KV transfer and gRPC traffic on a dedicated network.
- Keep runtime LoRA loading, tool servers and dev mode off in production.
- Remove hub, registry and cloud keys from serving runtimes; exclude any that remain from Ray propagation.
- Block instance metadata access and scope node roles to read-only model storage.
- Set request and media limits and media domain allowlists.
- Track NVIDIA bulletins and vLLM advisories and patch each serving pool promptly.
Standards and references
- vLLM: Security
- NVIDIA: Triton Inference Server Secure Deployment Considerations
- MITRE ATT&CK: Exploit Public-Facing Application
- NIST: SP 800-53 Rev. 5
- OWASP: Top 10 for LLM Applications 2025
- OWASP: API Security Top 10 2023
- MITRE ATLAS: AI Model Inference API Access
Related NHI Mgmt Group resources: Self-Hosted LLM Security Guide · Ray and AI Compute Cluster Security Guide · AI Infrastructure Workload Identity Guide · MLflow and Kubeflow Security Guide · AI Platforms Guide