Self-Hosted LLM Security Guide: Ollama, vLLM and LM Studio

Self-hosted inference servers such as Ollama, vLLM, LM Studio and llama.cpp were built for convenience on a trusted machine, and most of them ship with little or no authentication on their HTTP APIs. Teams run them to keep prompts and data in-house, to cut API costs or to experiment with open-weight models, then expose them on a network so colleagues, agents or applications can reach them. At that point an unauthenticated API that can pull, create and delete models, or a cluster port that accepts serialised Python objects, becomes an internet-facing service. This guide covers the exposure model of each server, how to put authentication and network controls in front of them, how to handle model files safely, and what to log and measure. It is written for platform engineers, security architects and AI engineering leads who run or approve self-hosted models.

Key takeaways

• Assume no authentication by default. Ollama documents no built-in API authentication, vLLM's --api-key protects only some path prefixes, and LM Studio's "Require Authentication" is a switch you must turn on.
• Binding is the first control. Ollama and llama.cpp bind to 127.0.0.1 by default; the risk starts when someone sets OLLAMA_HOST=0.0.0.0, runs the Docker image, or turns on "Serve on Local Network".
• Put a gateway in front and allowlist endpoints. A reverse proxy that authenticates callers and only forwards inference paths closes model management, metrics and operational endpoints in one step.
• Model files are code until proven otherwise. Pickle-based weights can execute on load; prefer safetensors or GGUF from known publishers, and inspect new models in an isolated zone before production.
• The inference host is a credential store. It often holds Hugging Face tokens, cloud roles and storage keys, so scope them tightly, keep them out of shared environment variables and restrict egress.

Why self-hosted LLMs are a distinct attack surface

A hosted model API is a managed service with identity, rate limiting and abuse monitoring built in. A self-hosted server is a process you operate, and its security is whatever you configure around it. Three properties make these servers different from an ordinary web application:

• Management and inference share one API: on Ollama the same port that answers /api/chat also accepts /api/pull, /api/create, /api/push and DELETE /api/delete. Anyone who can reach the port can change which models exist on the host.
• Expensive compute is the asset: GPU time is valuable, so an open endpoint is a target for free inference, resale and denial of service, the self-hosted form of LLMjacking. OWASP LLM10:2025 Unbounded Consumption covers the cost and availability side.
• Distributed inference trusts the network: vLLM's security documentation states that inter-node communication (PyTorch Distributed, KV cache transfer, tensor and pipeline parallel traffic) is insecure by default and must be protected by an isolated network.

The scale of exposure is measurable. Wiz reported more than 1,000 publicly exposed Ollama servers when it disclosed CVE-2024-37032 in June 2024, and Cisco Talos found over 1,100 using Shodan in 2025, with about 20% actively serving models and the rest still open to unauthorised model uploads or configuration changes, according to The Register's report of the research. The LLMjacking 2024 to 2026 page describes Pillar Security's Operation Bizarre Bazaar, which targeted exposed Ollama servers and OpenAI-compatible endpoints rather than stolen keys.

Default exposure by server

The table summarises what each project's own documentation says. Check the version you run, because defaults and flags change between releases.

Two details catch teams out. First, vLLM's documentation explicitly warns not to rely on --api-key alone because the SageMaker-style /invocations path runs the same inference unprotected. Second, Ollama's default CORS policy allows origins on 127.0.0.1 and 0.0.0.0, and widening OLLAMA_ORIGINS lets web pages in a user's browser call the local API; keep it at the default unless a specific local web application needs access.

Network placement and authentication

Choose a deployment pattern

Controls that apply to every pattern

• Bind narrowly: bind the inference process to loopback or a private interface and let only the proxy reach it. For vLLM set VLLM_HOST_IP to a specific address rather than relying on defaults.
• Authenticate at the gateway: terminate TLS and authenticate every caller at a reverse proxy (Ollama's FAQ shows forwarding to http://localhost:11434 with the Host header set). Use your identity provider for people and short-lived tokens or mTLS for applications and agents. See the AI Infrastructure Workload Identity Guide.
• Allowlist paths, not just ports: forward only the inference routes clients need (for example /v1/chat/completions and /v1/embeddings) and block model management, weight updates, metrics and tokeniser routes from general users.
• Turn on the native switch as well: enable LM Studio's "Require Authentication", set --api-key-file on llama.cpp and VLLM_API_KEY on vLLM as a second layer, with keys held in a secrets manager rather than shell history. See the Secrets Management Guide.
• Isolate cluster ports: keep vLLM KV transfer, data parallel and gRPC ports, the PyTorch TCPStore, and the Ray client port and dashboard on an isolated segment, and firewall everything else. The vLLM documentation treats the whole Ray cluster as one trust domain.
• Rate limit and cap requests: apply per-client quotas at the gateway, keep OLLAMA_MAX_QUEUE bounded, and lower VLLM_MAX_N_SEQUENCES (default 16384) for broadly shared deployments.

Model files, adapters and the supply chain

A self-hosted server is only as trustworthy as the weights and code it loads. Hugging Face's documentation explains that pickle, the default PyTorch serialisation format, runs opcodes on load and can execute arbitrary code, and that its Hub pickle import scan is not foolproof. MITRE ATLAS tracks this as AML.T0011.000 Unsafe AI Artifacts, and OWASP LLM03:2025 Supply Chain covers vulnerable models, adapters and conversion services.

• Prefer non-executable formats: safetensors and GGUF store tensors without code. Treat .bin, .pt and .pkl weights from outside your organisation as untrusted.
• Use a model allowlist: pull only approved publishers and pinned revisions or digests, and mirror approved models into an internal registry so production servers do not pull from the internet. Ollama pulls only over HTTPS; never use the API's insecure option outside a development registry.
• Inspect before you run: the joint NSA, CISA, FBI, NCSC-UK and partner guidance "Deploying AI Systems Securely" (April 2024) advises not running imported models straight in the enterprise environment, inspecting them in a secure zone, using AI-aware malicious code scanners and verifying signatures or checksums.
• Control adapter loading: vLLM's documentation says dynamic LoRA loading is not secure and should be limited to trusted administrators. See the Fine-Tuning and LoRA Security Guide.
• Protect caches and model directories: vLLM loads VLLM_CACHE_ROOT contents without integrity verification; Ollama stores models under OLLAMA_MODELS (on Linux /usr/share/ollama/.ollama/models by default). Restrict write access to the service account, and never copy cache artefacts from untrusted sources.
• Watch the server's own dependencies: the xinference PyPI compromise in April 2026 put a credential stealer inside releases 2.6.0 to 2.6.2 of an inference server package, which ran on import. Pin versions, use lockfiles with hashes and build images from a reviewed baseline. See the Open-Weight Model Security Guide and the AI Supply Chain and AI-BOM Guide.

Where identity fits

Inference hosts accumulate non-human credentials: a Hugging Face token to pull gated models, a cloud role for object storage, registry tokens, and API keys issued to client applications. vLLM's documentation warns that its Ray executor copies the driver's environment variables, which can include HF_TOKEN, to every worker by default, and recommends a denylist file (~/.config/vllm/ray_non_carry_over_env_vars.json), secrets injected from a manager, hidepid=2 on /proc and separate UIDs for driver and workers.

• Run as a dedicated, unprivileged service account: avoid running containers as root, give the account write access only to its model and cache directories, and no interactive login.
• Issue one credential per client: applications and agents that call the gateway each get their own token or workload identity, so usage can be attributed, throttled and revoked. Shared keys pasted into notebooks defeat all three.
• Scope cloud roles to the minimum: read-only access to the model bucket, no access to unrelated secrets, and metadata service hardening (for example IMDSv2 on AWS) so a compromised process cannot reuse the instance identity broadly.
• Restrict egress: production inference servers rarely need the internet; an egress allowlist limits both model pulls and credential exfiltration.

See the LLMjacking Guide for key handling on the client side.

Implementing a secure self-hosted LLM service

1. Discover what is running: scan internal ranges for the default ports (11434 for Ollama, 8080 for llama.cpp, and the ports your teams configure for vLLM and LM Studio) and check endpoint agents for the installed applications. Record owner, models and exposure for each. See the Shadow AI Discovery Guide.
2. Close the internet first: remove any public exposure immediately, then confirm with an external scan. This is the step with the highest risk reduction for the least effort.
3. Upgrade: bring every server to a current release; for example CVE-2024-37032 in Ollama was fixed in 0.1.34, and CVE-2025-32444 (CVSS 10.0, vLLM Mooncake integration) in vLLM 0.8.5.
4. Deploy the gateway in observe mode: route traffic through the proxy with authentication enabled but logging blocked paths rather than denying them for a short pilot, so you see which clients use which routes.
5. Enforce: switch the allowlist to deny, rebind the servers to loopback or a private interface, and enable native API keys as a second layer. Roll back by reverting the proxy rule set, not by re-exposing the server.
6. Move model intake into a pipeline: approved sources, format checks, scanning and checksum recording in a staging zone, then promotion to the internal mirror.
7. Harden the host and cluster: unprivileged service account, isolated inter-node network, Ray TLS where Ray is used, egress allowlist, secrets from a manager.
8. Test: confirm from an untrusted network segment that management, metrics and cluster ports are unreachable and that unauthenticated calls to every route fail. Repeat after every upgrade.

Small team versus large organisation

A small team can get most of the benefit with loopback binding, a single reverse proxy with SSO and an approved-models list. A large organisation should run a shared AI gateway in front of all self-hosted and hosted models, so authentication, quotas, logging and data controls are consistent, and should treat a team-run inference server outside that gateway as an exception that needs approval. See the AI Guardrails and AI Firewalls Guide.

Detection and measurement

• Gateway logs: caller identity, route, model, token counts, latency and status code for every request. The joint "Deploying AI Systems Securely" guidance also recommends logging inputs, outputs and errors and alerting on high-frequency or repetitive inputs; decide what prompt content you can retain under your privacy rules.
• Model inventory changes: alert on any call to Ollama's /api/pull, /api/create, /api/delete or blob upload, on vLLM /update_weights, and on new files in model directories outside the promotion pipeline.
• Host and network signals: new listeners on 0.0.0.0, connections to cluster ports from outside the inference segment, unexpected outbound connections and child processes spawned by the inference service.
• Capacity and cost: sustained GPU saturation outside business patterns, queue overflows (Ollama returns 503 when OLLAMA_MAX_QUEUE is full) and per-client usage spikes.

Framework mapping

For ISO/IEC 27001:2022, the most relevant Annex A controls are 8.9 Configuration management, 8.20 Networks security, 8.22 Segregation of networks, 8.15 Logging and 8.16 Monitoring activities. See the MITRE ATLAS Guide and the OWASP Top 10 for LLM Applications Guide.

Common failure patterns

• "It is only on the internal network": a team server bound to 0.0.0.0 without authentication is reachable from every compromised laptop, and often ends up behind a port forward or a cloud security group opened for a demo. The exposed populations counted by Wiz and Cisco Talos are mostly servers like this.
• Trusting a partial API key: setting VLLM_API_KEY and assuming the server is protected, while /invocations, /metrics and operational endpoints stay open.
• Running the container defaults in production: the Ollama Docker image listening on all interfaces as root, with a published port, turns any file-write bug such as CVE-2024-37032 into host compromise.
• Exposing cluster plumbing: KV transfer, ZeroMQ, TCPStore or Ray ports on a routable network. CVE-2025-32444 showed how pickle deserialisation on sockets listening on all interfaces becomes unauthenticated remote code execution.
• Pulling any model a user asks for: no allowlist, pickle weights from unknown publishers and model management left open to every client, so anyone can load an untrusted artefact on the shared GPU host.
• Secrets in the environment: HF_TOKEN and cloud keys exported in the driver shell and copied to every worker, or broad instance roles that a stealer like the one in the xinference releases can reuse.

Practitioner checklist

• Inventory every Ollama, vLLM, LM Studio and llama.cpp instance with owner, version, bind address and models.
• Confirm by external scan that no inference server is reachable from the internet.
• Bind inference processes to loopback or a private interface and front shared servers with an authenticating, TLS-terminating gateway.
• Allowlist inference routes at the gateway and block model management, weight update, metrics and debug routes for general clients.
• Enable native authentication as a second layer: LM Studio "Require Authentication", llama.cpp --api-key-file, VLLM_API_KEY.
• Keep inter-node, gRPC, KV transfer and Ray ports on an isolated network segment; never set VLLM_SERVER_DEV_MODE=1 in production.
• Source models only from an internal mirror of approved, pinned artefacts; prefer safetensors or GGUF and inspect new models in a staging zone.
• Run as an unprivileged service account with secrets from a manager, scoped cloud roles and an egress allowlist.
• Alert on model pulls, creates, deletes and weight updates, new listeners and per-client usage spikes.
• Patch to current releases within a set window and re-test exposure after every upgrade.

Standards and references

• NSA, CISA, FBI, NCSC-UK and partners: Deploying AI Systems Securely
• OWASP: Top 10 for LLM Applications 2025
• MITRE ATLAS: AI Model Inference API Access
• vLLM: Security
• Ollama: FAQ
• LM Studio: Server Settings
• llama.cpp: Server README
• Hugging Face: Pickle Scanning

Related NHI Mgmt Group resources: AI Model Security Guide · Open-Weight Model Security Guide · LLM Security Guide · AI Security Maturity Model · Hugging Face Security Guide