Self-Hosted LLMs vs Hosted LLM APIs: Threat Models, Data Residency and Operating Cost

Running a model on your own infrastructure does not make it more secure than calling a provider's API; it moves the risk from a credential and a contract to servers, model files and a team you must staff. Teams choose self-hosting to keep prompts in-house, meet residency rules or control cost, and choose hosted APIs for capability and speed. Both choices are defensible, and both fail in predictable ways. This guide compares the two as architectures: the threat model of each, where data goes and how long it stays, what is exposed (keys versus infrastructure), operational burden, and a decision table with a worked scenario. It is for security architects, AI platform owners and risk teams. How to harden an inference server is in the Self-Hosted LLM Security Guide, and how to protect provider keys is in the LLMjacking Guide; this page does not repeat them.

Key takeaways

• Short answer: the deciding factor is usually data obligations and team capacity, not raw security. A well-configured hosted API with zero retention and private networking often carries less risk than a self-hosted server run by a team without platform security skills.
• Hosted risk concentrates in identity and contract. The attack surface is the credential that calls the API and the provider's retention, abuse monitoring and processing location, all of which are configurable and must be checked per model.
• Self-hosted risk concentrates in infrastructure. Unauthenticated endpoints, insecure inter-node traffic and model files that execute code on load become your problem; vLLM, for example, documents inter-node communication as insecure by default.
• "Hosted" is not one retention model. Providers document different defaults for caching, abuse monitoring and human review, and these vary by model and feature within the same platform.
• Most organisations end up hybrid. A gateway that routes sensitive classes of data to self-hosted models and everything else to hosted APIs, with one identity and logging layer, is the common end state.

What each option is

Self-hosted LLM: you run model weights on hardware you control, on premises or on cloud GPU instances in your own account, using an inference server such as llama.cpp, Ollama or vLLM. You own the network exposure, authentication, model provenance, patching, scaling and logging. Prompts never leave your boundary unless your own systems send them.

Hosted LLM API: you send prompts over HTTPS to a provider that runs the model. This covers model developers' own APIs and cloud platforms that offer many models through one service. You own the credential, network path, data classification and configuration; the provider owns the model runtime and documents what it stores, for how long, and where it processes requests.

A third pattern sits between them: open-weight models deployed as managed endpoints in your cloud account. Treat it as hosted for runtime and data handling, and as self-hosted for model provenance.

Side-by-side comparison

Threat model: self-hosted LLMs

The self-hosted threat model is that of an internal web service with an expensive, unusual workload behind it.

• Unauthenticated or partly authenticated endpoints: vLLM's own security guidance says its --api-key protects only the /v1, /v2, /inference and /cohere prefixes, leaves endpoints such as /tokenize, /pause and /update_weights without authentication, and recommends a reverse proxy that allowlists only intended endpoints. Attackers who reach an open endpoint use it for free inference (ATLAS AML.T0040 AI Model Inference API Access) or to change which models are served.
• Insecure cluster traffic: vLLM documents inter-node communication (PyTorch Distributed, KV cache transfer, parallel traffic) as insecure by default, unencrypted and accepting connections without checks, and notes that the PyTorch TCPStore listens on all interfaces. Multi-node inference needs an isolated network.
• Model files as code: Hugging Face warns that unpickling can execute arbitrary code and that its Hub pickle scan "is not 100% foolproof". A tampered model from a public repository (ATLAS AML.T0010.003 Model Repository) can compromise the GPU host on load.
• Server-side request forgery: multimodal servers fetch media from URLs. vLLM provides --allowed-media-domains and recommends VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 to stop redirects bypassing the allowlist.
• Credentials on the host: inference hosts often hold model repository tokens, storage keys and cloud roles (ATLAS AML.T0055 Unsecured Credentials), making them a pivot point.
• Data in your own logs: self-hosting keeps prompts in your boundary, but request logging, traces and debugging dumps can retain them indefinitely unless you design retention.

Failure modes: a test server bound to all interfaces and forgotten; a GPU cluster with management ports reachable from user networks; an engineer pulling an unvetted pickle-format checkpoint into production; nobody on call for the inference tier. Server-by-server defaults and fixes are in the Self-Hosted LLM Security Guide and Open-Weight Model Security Guide.

Threat model: hosted LLM APIs

The hosted threat model is that of any high-value SaaS API reached with a credential.

• Key theft and cost harvesting: long-lived API keys leaked in code, CI or client applications are used to run inference at your expense (ATLAS AML.T0034 Cost Harvesting). Per-token pricing means abuse becomes a bill, and OWASP LLM10:2025 Unbounded Consumption covers the cost and availability impact. See How to set spend limits on LLM API keys.
• Retention you did not choose: caching, stored conversations, abuse monitoring and grounding features each have their own retention, and some cannot be disabled. Sensitive data sent under default settings may be held for days to weeks.
• Processing location drift: global or cross-region deployment types can process prompts outside the geography where data is stored, which matters for residency commitments.
• Exfiltration through the API: an insider or compromised workload can send sensitive data to an external model (ATLAS AML.T0024 Exfiltration via AI Inference API), so egress to model endpoints should go through a controlled path.
• Provider dependency: model versions are retired, behaviour changes between versions, and a provider incident affects every customer at once. See the AI Vendor Breach Response guide.

Failure modes: one shared key across teams with no attribution; personal accounts on consumer tiers used for company data; global deployment types chosen for capacity without checking residency; abuse monitoring left at default for regulated data without a documented decision.

Residency and retention on hosted platforms

Retention and processing controls differ by platform, model and feature, so check each model you enable. The three cloud AI platforms below are listed alphabetically, and the order is not a ranking. The facts are taken from each provider's documentation as checked in November 2026.

Amazon Bedrock

AWS documents a model deployment account per model provider in each Region, operated by the Bedrock team, and states that model providers have no access to Bedrock logs or customer prompts and completions. Bedrock offers an account-level, per-Region data retention setting: in mode none, no request or response data is written to durable storage or shared with the model provider. Under the default mode AWS may retain data for safety and abuse prevention, with retention depending on the model's terms, and the model provider does not receive it; for models that require the aws_review mode, prompts and completions are retained within the AWS boundary for up to 30 days and may be reviewed by AWS. With cross-region inference, retained data is stored in the Region where the request is processed. Availability: some models require human review and are unavailable under modes none or default. Configuration depth is in the Amazon Bedrock Security Guide.

Google Cloud (Vertex AI, now documented under Gemini Enterprise Agent Platform)

Google states it will not use customer data to train or fine-tune models without prior permission or instruction. Implicit in-memory caching is on by default for supported Gemini models, held in RAM for no longer than 24 hours, and an administrator can turn off all caching at project level through the cacheConfig setting. Zero data retention for Google models generally requires request-response logging left off, store = false in Interactions API requests, and caching disabled. Grounding with Google Search stores derived queries for up to three days and grounding with Google Maps for 30 days, neither of which can be disabled. With the global endpoint, requests can be served from any region. Availability: zero retention for some partner models is off by default and requires per-project approval. Configuration depth is in the Google Vertex AI Security Guide.

Microsoft Azure (Azure OpenAI in Foundry)

Microsoft states that prompts, completions and training data are not used to train base models without permission and are not available to other customers or model providers. Data is stored only for features you use (for example files, vector stores, stored completions and batch), at rest in the same geography with AES-256 encryption by default. Standard deployments process prompts within the chosen geography; DataZone deployments within a defined data zone such as the EU; Global deployments in any geography where the model is deployed, while stored data stays in your geography. Abuse monitoring stores flagged prompts for human review in the resource's geography. Availability: modified abuse monitoring, which removes storage and human review, requires approval under Limited Access criteria; the resource's ContentLogging value shows false once it applies. Configuration depth is in the Azure OpenAI and Azure AI Foundry Security Guide.

For any hosted provider, retention should be a written decision per data class, as covered in Zero Data Retention for AI.

Key exposure versus infrastructure exposure

Hosted access is safest when no static key exists: see Workload Identity Federation vs Static Cloud Keys.

Operational burden and cost

• Staffing: self-hosting needs people who can run GPU infrastructure, patch inference servers quickly, vet models and respond out of hours. If that team does not exist, the security benefit of keeping data in-house is offset by the exposure of an unmanaged service.
• Utilisation: self-hosted cost is mostly fixed, so it suits steady, high-volume workloads; spiky or experimental use is usually cheaper per token on an API.
• Capability and change: hosted platforms add models and features quickly, which is useful but also means new defaults to review. Self-hosted stacks change only when you upgrade, which is predictable but needs a patch process.
• Evidence for auditors: hosted platforms provide attestations for the runtime, while self-hosting requires you to produce evidence for every layer.

Choosing: decision table

Small organisations rarely benefit from self-hosting for security reasons alone; a hosted platform in their cloud account with zero retention and workload identity is usually the lower-risk choice. Large organisations tend to run both and need a gateway to enforce routing: see the AI Gateway and LLM Gateway Guide.

Worked scenario: insurer with claims and marketing workloads

A European insurer has two AI use cases. Claims handling summarises medical reports, which the insurer's legal team classifies as special category data that must not leave its own environment. Marketing drafts campaign copy from public material and wants the most capable models.

• Claims: a self-hosted open-weight model on GPU nodes in the insurer's own cloud account, nodes on an isolated subnet, vLLM behind an authenticating proxy that allows only inference paths, safetensors-only model intake with hash verification, request logs kept for 30 days with prompt bodies redacted. A platform team of three owns patching and on-call.
• Marketing: a hosted API through the company's cloud platform, regional or data-zone deployment in the EU, retention set to the minimum the chosen model allows, workload identity for the application and a monthly spend cap.
• Shared layer: an internal gateway authenticates every caller, tags requests by data class, refuses to route claims data to the hosted endpoint, and sends usage logs to the SIEM.

The figures in this scenario are illustrative.

Where identity fits

Every model call is made by an identity. On hosted platforms that is an API key, a cloud role or a managed identity; on self-hosted platforms it is whatever your proxy authenticates. Give each application its own non-human identity, prefer federated or managed identities over static keys, and keep a model-access inventory that names the owner of each identity. Inference hosts also hold identities of their own (repository tokens, storage roles) that need the same lifecycle.

How to implement: phased approach

1. Classify use cases (weeks 1 to 3): list AI workloads with the data classes they send, residency obligations and required model capability.
2. Set hosted baselines first (weeks 2 to 6): approved platforms and regions, retention mode per data class, workload identity, private endpoints, spend caps. Test by sending synthetic sensitive data and confirming logging and storage settings.
3. Stand up self-hosting only where the table says so (weeks 6 to 14): isolated network, authenticating proxy with endpoint allowlist, model intake process, on-call rota. Pilot with one workload; keep the hosted fallback available for rollback where data rules allow.
4. Put a gateway in front of both: route by data class, enforce identity, log centrally.
5. Review quarterly: provider retention terms, newly enabled models, deployment types and self-hosted patch levels.

Detection and measurement

Log every model call with caller identity, model, endpoint, data class tag and token counts. On self-hosted platforms add endpoint path, model load events and GPU utilisation; on hosted platforms add provider audit logs for model enablement and configuration changes.

• Static key share: percentage of model calls made with static API keys rather than federated or managed identities.
• Retention conformance: share of hosted deployments whose retention and deployment type match the documented decision for their data class.
• Unauthenticated endpoint count: self-hosted inference endpoints reachable without authentication, from scans; target zero.
• Model intake compliance: share of self-hosted models in non-executable formats with recorded provenance and hash.
• Spend anomaly time to detect: time from abnormal usage to alert, for both hosted keys and self-hosted GPU use.

Common failure patterns

• Self-hosting for privacy without a platform owner: the server is exposed, unpatched and logs every prompt to disk, which is worse than a configured hosted service.
• Assuming hosted means no retention: default caching, abuse monitoring or grounding retention left in place for regulated data.
• Global deployment types chosen for capacity: processing moves outside the committed geography without anyone recording the decision.
• Shared static keys: one key in many repositories, no attribution and no spend cap, as described in the LLMjacking Guide.
• Unvetted model files: pickle checkpoints loaded on production GPU hosts that also hold cloud credentials.
• Two estates, no common layer: separate logging and identity for hosted and self-hosted calls, so nobody sees total usage or data flows.

Framework mapping

Decision checklist

• Which data classes will each workload send, and may any of them leave your environment?
• For hosted use, which retention mode, deployment type and region apply to each enabled model?
• For self-hosted use, who owns patching, model intake and on-call?
• Is model access granted through workload identities rather than static keys?
• Does one gateway or logging layer see both hosted and self-hosted calls?

Practitioner checklist

• Record a hosting decision per AI workload against data class, residency and capability.
• Set hosted retention to the minimum the model allows for sensitive classes, and document exceptions.
• Prefer regional or data-zone deployment types where residency is committed.
• Replace static provider keys with federated or managed identities; cap spend per application.
• Place every self-hosted server behind an authenticating proxy that allows only inference paths.
• Isolate multi-node inference traffic on its own network.
• Accept only non-executable model formats with recorded hashes into production.
• Route all model traffic through one gateway and log identity, model and data class.
• Re-check provider retention and deployment documentation quarterly.

Standards and references

• NIST: SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations
• OWASP: Top 10 for LLM Applications 2025
• MITRE: ATLAS
• vLLM: Security guide
• Hugging Face: Pickle scanning and security
• AWS: Amazon Bedrock data retention
• Google Cloud: Zero data retention
• Microsoft Learn: Data, privacy and security for Azure OpenAI

Related NHI Mgmt Group resources: Self-Hosted LLM Security Guide, LLMjacking Guide, Zero Data Retention for AI, Model Supply Chain Guide, Third-Party AI Risk Guide.