Running LLMs Locally Explained: Tools, Hardware, Formats and Memory Sizing

Running an LLM locally means downloading a model's weights and generating text on your own computer or server, so prompts and outputs are processed on hardware you control instead of a provider's cloud API. The most widely used local runtimes are llama.cpp, LM Studio, Ollama and vLLM, and the main constraint is memory: a model only runs well if its weights, plus its working cache, fit in GPU memory or fast system RAM. This explainer covers what each runtime is, what hardware the projects themselves state they need, how quantised model files such as GGUF reduce memory, how to estimate whether a model will fit, and which settings change privacy and exposure. It is written for learners, developers and IT teams evaluating local models.

Last checked: 2026-11-15. Runtimes are listed alphabetically; order is not a ranking.

This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.

Key takeaways

• Local means the weights run on your hardware. You need an open-weight model file and a runtime; a closed model available only through an API cannot be run locally.
• Memory decides what fits. Weight size is roughly parameters multiplied by bits per weight, plus a context (KV) cache that grows with context length and parallel requests.
• Quantisation trades precision for size. GGUF quantisation types range from 8-bit down to about 1.56 bits per weight, as listed in the Hugging Face GGUF documentation.
• Each runtime documents its own hardware support. llama.cpp, LM Studio, Ollama and vLLM publish different operating system, CPU and GPU requirements, summarised in the tables below.
• Default network binding matters. Ollama and llama-server bind to 127.0.0.1 by default; changing the bind address exposes the model API to other machines.

What does running an LLM locally mean?

A large language model is a file (or set of files) of numeric weights. A runtime, also called an inference engine, loads those weights into memory and repeatedly predicts the next token. Running locally means both parts sit on a laptop, workstation or server you operate. Typical reasons are offline use, keeping prompts on the device, predictable cost, and experimentation with open-weight models.

• Model weights: published by the model's developer, commonly on Hugging Face, under the developer's licence. Licences differ by model, so check each model card before commercial use.
• Model file format: GGUF for llama.cpp, LM Studio and Ollama; safetensors and other formats for vLLM and Python frameworks.
• Runtime: the program that loads the file and serves a chat interface, command line or HTTP API.
• Client: a chat app, IDE plugin or your own code that calls the runtime, often through an OpenAI-compatible API.

For the trade-offs between running models yourself and calling a hosted API, see the Self-Hosted LLMs vs Hosted LLM APIs guide, and for what open weights do and do not include, the Open-Weight Model Security Guide.

The main local LLM runtimes

As of 2026-11-15, from each project's own documentation.

llama.cpp

llama.cpp is the C/C++ inference project built on the ggml library. Its README states support for 1.5-bit to 8-bit integer quantisation, Apple silicon via ARM NEON, Accelerate and Metal, AVX, AVX2, AVX512 and AMX on x86, custom CUDA kernels for NVIDIA GPUs, AMD GPUs via HIP, Vulkan and SYCL backends, and CPU plus GPU hybrid inference for models larger than total VRAM. The bundled llama-server provides OpenAI-compatible routes, parallel decoding and multimodal support.

LM Studio

LM Studio is a desktop application with a graphical model browser, a chat interface and a Developer tab that starts a local server, also available through the lms CLI. Its documentation states it can operate entirely offline once model files are downloaded. The server exposes LM Studio's REST API, TypeScript and Python SDKs, and OpenAI-compatible and Anthropic-compatible endpoints. Server settings include Server Port, Require Authentication, Serve on Local Network, Enable CORS and Just in Time Model Loading.

Ollama

Ollama packages models and a server behind a short CLI and a desktop app. Its FAQ states that it binds 127.0.0.1 port 11434 by default, that the default context window is 4096 tokens (changed with OLLAMA_CONTEXT_LENGTH), and that models are stored in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows. Ollama also offers optional cloud-hosted models; the FAQ states local use is not seen by Ollama and describes a disable_ollama_cloud setting (or OLLAMA_NO_CLOUD=1) for local-only mode.

vLLM

vLLM is a Python serving engine aimed at GPU servers. Its installation guide states Linux only (no native Windows support; WSL is suggested), Python 3.11 to 3.14, and NVIDIA GPUs with compute capability 7.5 or higher. It documents AMD ROCm and Intel XPU support, and Apple silicon acceleration through a community-maintained vLLM-Metal plugin. It is typically used where many users share one model on a server rather than on a single laptop.

What hardware do you need to run an LLM locally?

As of 2026-11-15, as stated by each project. Where a project does not state a figure, the cell says so.

• Unified memory (Apple silicon): CPU and GPU share one memory pool, so a Mac's total RAM sets the practical model size limit.
• Discrete GPUs: VRAM sets the limit for full-speed inference. llama.cpp's hybrid mode can place some layers in system RAM, which runs but is slower.
• CPU only: llama.cpp and Ollama document CPU inference (Ollama: x86 Macs run CPU only); check each runtime's own CPU guidance.

How quantisation and GGUF make models smaller

Models are usually trained and published at 16-bit precision (F16 or BF16). Quantisation stores each weight with fewer bits. The Hugging Face GGUF documentation describes GGUF as a binary format that stores tensors plus a standard set of metadata, developed by the llama.cpp author for fast loading, and lists these quantisation types (as of 2026-11-15):

Lower bits mean a smaller file and less memory, at some cost to output quality; the size of that cost varies by model and task. File names on Hugging Face usually carry the type, for example a "Q4_K_M" suffix. Hugging Face also provides a GGUF filter on its model search and a conversion Space for producing GGUF files.

How much memory does a local LLM need?

A working estimate has three parts. This is arithmetic from the bits-per-weight figures above, not a figure published by any runtime, so treat it as a starting point and check actual usage.

1. Weights: parameters multiplied by bits per weight, divided by 8, gives bytes. An 8 billion parameter model at 16 bits is about 16GB; at Q4_K (4.5 bits) about 4.5GB; at Q8_0 about 8GB.
2. Context (KV) cache: grows with context length. Ollama's FAQ states required RAM scales with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH, and that 4 parallel requests at 2K context behave like an 8K context.
3. Overhead: the runtime, operating system and other applications need headroom.

Weights only, as of 2026-11-15, calculated from the GGUF bits-per-weight values; add KV cache and overhead.

Ways to reduce memory use

• Smaller quantisation: move from Q8_0 to Q4_K, or lower, and test quality on your own tasks.
• Shorter context: lower the context length setting if you do not need long documents.
• KV cache quantisation: Ollama's OLLAMA_KV_CACHE_TYPE accepts f16 (default), q8_0 (about half the memory of f16) and q4_0 (about a quarter), when Flash Attention is active.
• Fewer parallel requests: Ollama's OLLAMA_NUM_PARALLEL defaults to 1.
• Unload idle models: Ollama's keep_alive parameter and OLLAMA_KEEP_ALIVE control how long a model stays loaded; LM Studio offers auto-unload for JIT-loaded models.

How do you choose a runtime?

Factual criteria only; each runtime documents different strengths, and the right fit depends on your environment.

Is running an LLM locally private and secure?

Local inference keeps prompts on your hardware by design, but privacy depends on configuration. Ollama's FAQ states it does not see prompts when you run locally, and that cloud-hosted models process prompts to provide that service. LM Studio's app terms refer data questions to its separate privacy policy.

• Bind address: Ollama (127.0.0.1:11434) and llama-server (127.0.0.1) are local-only by default. Setting OLLAMA_HOST to 0.0.0.0, or enabling LM Studio's Serve on Local Network, makes the API reachable by other machines.
• Authentication: LM Studio has a Require Authentication setting; llama-server documents an --api-key option (default: none). Turn authentication on before exposing any server.
• CORS: llama-server's --cors-origins default is *; Ollama allows 127.0.0.1 and 0.0.0.0 origins by default and adds others through OLLAMA_ORIGINS.
• Agent tools: llama-server's experimental built-in tools (including shell execution and file write) are off by default and documented with a "do not enable in untrusted environments" warning.
• Model provenance: download from the publisher's own repository and check the licence and file type.

For hardening, exposure checks and monitoring, see the Self-Hosted LLM Security Guide; for laptops and phones, the On-Device AI Security Guide; and for connecting local tools to agents, the Local MCP Server Security Guide.

How to get started: a step-by-step path

1. Check your memory: note total RAM (Apple silicon) or VRAM (discrete GPU) and use the table above to pick a model size that leaves headroom.
2. Pick a runtime: match it to your operating system using the hardware table.
3. Choose a model and licence: read the model card on the publisher's page for licence, context length and intended use.
4. Choose a quantisation: start around Q4_K or Q5_K for laptops, Q8_0 where memory allows.
5. Test locally first: keep the default local bind address while you evaluate quality and speed.
6. Measure: watch memory use, tokens per second and quality on your own prompts; adjust context length and quantisation.
7. Only then share: if others need access, turn on authentication, restrict network exposure and follow the security guide linked above.

Common mistakes

• Sizing only for weights: long contexts and parallel requests add KV cache memory, so a model that loads can still run out of memory mid-conversation.
• Assuming the default context is the model's maximum: Ollama's documented default is 4096 tokens regardless of what the model supports.
• Opening the server to the network without authentication: changing the bind address is a one-line setting with large exposure effects.
• Ignoring the model licence: the runtime's licence does not cover the model weights.
• Using unofficial re-uploads: prefer the publisher's own files or well-documented conversions.

Is running an LLM locally free?

llama.cpp, Ollama and vLLM are open source under MIT or Apache 2.0 licences. LM Studio's terms grant a licence for personal and internal business purposes and state some features may be offered for a fee. Model weights carry their own licences. Hardware and electricity are the main ongoing costs.

Practitioner checklist

• Record RAM or VRAM and estimate weights plus KV cache before downloading a model.
• Confirm the runtime supports your operating system and GPU from its own documentation.
• Read the model card for licence, context length and intended use.
• Choose a quantisation level and test quality on your own prompts.
• Keep the server on 127.0.0.1 unless remote access is required.
• Turn on authentication and restrict CORS before any network exposure.
• Disable optional cloud features where strict local-only use is required (Ollama local-only mode).
• Keep runtimes updated from the official release channels.

Standards and references

• ggml-org: llama.cpp README
• ggml-org: llama.cpp server README
• LM Studio: System Requirements
• LM Studio: Server Settings
• LM Studio: App Terms
• Ollama: FAQ
• Ollama: Hardware support
• Ollama: OpenAI compatibility
• vLLM: GPU installation
• Hugging Face: GGUF

Related NHI Mgmt Group resources: Self-Hosted LLM Security Guide, Self-Hosted LLMs vs Hosted LLM APIs, Open-Weight Model Security Guide, On-Device AI Security Guide, Hugging Face Security Guide.