vLLM vs Ollama vs llama.cpp: Which to Self-Host?

vLLM, Ollama, and llama.cpp are three open-source LLM inference servers for self-hosted deployment: vLLM serves the most concurrent users per GPU, Ollama reaches a working private model fastest, and llama.cpp runs on the widest hardware, including CPU-only machines. They optimize different things, so the choice follows your constraint rather than a benchmark leaderboard. This guide compares all three on batching, setup effort, hardware fit, OpenAI-compatible APIs, multi-GPU behavior, and air-gapped installability, then gives a four-question triage you can run against your own deployment.

What is an LLM inference server, and what do these three do?

An inference server is the software layer that loads open-weight model files onto hardware you control and answers generation requests over an HTTP API. It is the serving tier of a self-hosted stack, not a model itself.

• vLLM is an Apache 2.0 licensed, GPU-first serving engine that came out of UC Berkeley. It targets high-throughput multi-user serving using PagedAttention for key-value cache management and continuous batching to keep the GPU saturated. It loads Hugging Face safetensors weights and ships an OpenAI-compatible HTTP server.
• Ollama is an MIT-licensed local model runner written in Go that wraps a GGUF inference engine (historically llama.cpp, plus its own GGML-based engine for newer model families) behind a one-command experience. ollama run <model> downloads, loads, and serves a model with no configuration file.
• llama.cpp is an MIT-licensed C and C++ inference engine built on the GGML tensor library and the GGUF quantized model format. It compiles to a dependency-free binary, runs on CPU, Apple Silicon, NVIDIA, AMD, Intel, and Vulkan-capable hardware, and exposes an HTTP API through its llama-server binary.

One relationship explains most benchmark confusion: Ollama and llama.cpp are close relatives running the same GGUF quantized weights, while vLLM is an architecturally different engine aimed at a different workload.

How do vLLM, Ollama, and llama.cpp compare side by side?

Dimension	vLLM	Ollama	llama.cpp
Primary strength	Throughput under concurrency	Fastest time to a working model	Runs anywhere, including CPU
Runtime	Python / PyTorch / CUDA	Go binary wrapping a GGML engine	Self-contained C/C++ binary
Model format	Hugging Face safetensors (GGUF support limited)	GGUF	GGUF
Hardware	NVIDIA CUDA first; ROCm, Intel, TPU, Neuron backends vary in maturity	GPU with automatic CPU fallback	CPU, Metal, CUDA, ROCm, Vulkan, SYCL
CPU-only viability	Poor	Workable for small models	Excellent, its original design target
Concurrency model	Continuous batching plus PagedAttention	Parallel slots, tuned by environment variables	Parallel slots in llama-server
Multi-GPU	Tensor and pipeline parallelism, multi-node via Ray	Layer split across GPUs (capacity, not speed)	Layer or row split across GPUs
Model larger than VRAM	Possible via CPU weight offload, at a steep throughput cost	Yes, spills to CPU RAM	Yes, partial offload by layer count
Quantization	AWQ, GPTQ, FP8, INT8/INT4 (GGUF experimental)	GGUF K-quants and IQ-quants	GGUF K-quants and IQ-quants, down to about 1.5-bit
OpenAI-compatible API	Yes, comprehensive	Yes, partial, plus its own native API	Yes, in llama-server
Setup effort	Highest (Python environment, CUDA, tuning)	Lowest (single install, one command)	Moderate (compile with the right backend flags)
Air-gapped install	Container image or pinned wheels plus weight files	Transfer blob store or build from local GGUF	Vendor source or binary plus GGUF file
Best fit	Shared production service on GPUs	Prototypes, single-team tools, dev laptops	CPU servers, edge, constrained or mixed hardware

When should you choose vLLM?

Choose vLLM when a self-hosted model must serve many people at once on GPU hardware, the normal shape of a departmental or institution-wide deployment. vLLM’s advantage is not raw single-request speed; it is how many simultaneous requests one GPU carries before latency degrades.

Two mechanisms produce that advantage:

• PagedAttention stores the attention key-value cache in fixed-size, non-contiguous blocks, the way an operating system pages virtual memory. Naive serving reserves one contiguous cache slab per request sized for the worst case, wasting GPU memory on fragmentation and padding. Paging the cache means more concurrent sequences fit in the same VRAM.
• Continuous batching schedules at the iteration level rather than the request level. A finished sequence leaves the batch and a queued one joins immediately, instead of every request waiting for the slowest member of a static batch.

The cost is operational weight. vLLM expects a maintained Python and CUDA environment, preallocates a large share of GPU memory at startup, assumes the weights and the KV cache live in VRAM, and needs real tuning at your context length and concurrency. It can offload part of the weights to system RAM when a model does not fit, but that path exists as a fallback and gives up much of the throughput you chose vLLM for. That investment pays off when the GPU is a shared production asset, and it is overkill for one analyst on a workstation. Size the card first using the air-gapped LLM hardware and VRAM sizing guide, because fitting the model in GPU memory is what the rest of vLLM’s design assumes.

When should you choose Ollama?

Choose Ollama when the priority is a private model running quickly with minimal operational surface: a proof of concept, an internal developer tool, one team’s assistant, or a laptop environment that mirrors production. Installation is one package, model loading is one command, and Ollama handles GPU offload with automatic CPU fallback, so a model larger than your VRAM still runs, slowly, instead of failing outright.

Ollama stops being the right answer at scale. Its parallelism is set by environment variables (parallel requests and loaded models) rather than a scheduler built for saturation, so per-GPU throughput under sustained concurrent load sits well below vLLM’s. Multi-GPU in Ollama means splitting a model’s layers across cards to fit something bigger, not multiplying throughput. Treat Ollama as the fastest path to a working system and the natural development-side twin of a vLLM production tier: both speak an OpenAI-compatible API, so application code rarely changes between them.

When should you choose llama.cpp?

Choose llama.cpp when hardware is the binding constraint. It is the only one of the three genuinely designed for CPU inference, and it spans the widest range of accelerators: Apple Metal, NVIDIA CUDA, AMD ROCm, Intel SYCL, and Vulkan on almost everything else. Three on-premise situations point straight to it:

1. No GPU budget or no GPU availability. A quantized small model on a CPU server is slow but real, and for batch document processing or a low-traffic internal tool that is often enough.
2. Model larger than VRAM. Partial offload, a chosen number of layers on GPU and the rest in system RAM, lets a 24 GB card run a model it could never hold outright, trading tokens per second for capability.
3. Edge and constrained deployment. A branch office, a factory floor terminal, a field device, or an ARM box gets a single compiled binary with no Python runtime and no package manager, which is also the easiest artifact to review and vendor into a locked-down environment.

llama.cpp’s llama-server does support continuous batching and parallel slots, so it is not single-user only. At high concurrency on data-center GPUs, though, a paged-attention engine remains the more appropriate tool.

Do all three offer an OpenAI-compatible API?

Yes. vLLM, Ollama, and llama.cpp each expose an OpenAI-compatible HTTP endpoint, which is the single most important portability fact in on-premise LLM architecture. vLLM ships the most complete implementation (chat completions, completions, embeddings, streaming, tool and function calling, structured output). llama.cpp’s llama-server provides a compatible chat completions endpoint alongside its native API. Ollama offers an OpenAI-compatible layer in addition to its own REST API on port 11434, with coverage that is deliberately partial.

The practical consequence: write your retrieval and orchestration layer against the OpenAI-compatible interface and keep the base URL in configuration. You can start on Ollama, move to vLLM as concurrency grows, and fall back to llama.cpp at an edge site without rewriting the application. Verify the specific features you depend on, because “OpenAI-compatible” is a spectrum, and tool calling, structured output, and embeddings are the usual gaps.

Which inference server installs cleanly in an air-gapped network?

All three run air-gapped, but they differ in how much work it takes to get the artifacts inside the perimeter. Each assumes internet access on its default path, so plan the transfer explicitly.

• llama.cpp is the easiest to air-gap: one repository or prebuilt binary plus one GGUF file, with no package ecosystem to mirror.
• Ollama needs the model out of its public registry. Pull on a connected staging machine and transfer the blob store, or convert a GGUF you already hold and register it locally with a Modelfile. The server itself is a single binary.
• vLLM is the heaviest transfer: either a pinned container image or a Python wheel set with matched CUDA and PyTorch versions, plus the safetensors weights. Container images are the sane route, since resolving Python dependencies offline is where these installs usually fail.

Whichever engine you pick, the offline lifecycle needs the same discipline covered in the on-premise LLM deployment architecture guide: validate in a staging enclave, transfer signed artifacts through approved media, load the new version alongside the old one, and promote only after it passes your own evaluation.

How do you decide? A four-question triage

Answer these in order. The first constraint that binds usually decides the engine.

1. Do you have GPUs, and does the model fit in VRAM? No GPU, or a model bigger than the card, points to llama.cpp with partial offload. Yes to both keeps vLLM in play.
2. What is sustained concurrency at peak? A handful of users at a time is fine on Ollama or llama.cpp. Dozens of simultaneous requests against a shared GPU is where vLLM’s paged KV cache and continuous batching pay for themselves.
3. Who operates it on day two? A team without a dedicated CUDA and Python environment owner will get more reliable service from Ollama or a llama.cpp binary than from a mistuned vLLM cluster.
4. How constrained is the environment? A strict air gap, unusual hardware, or a locked-down base image favors the smallest artifact, which is llama.cpp.

The mental model worth keeping: vLLM buys throughput with operational complexity, Ollama buys simplicity with a throughput ceiling, and llama.cpp buys hardware reach with raw speed. There is no wrong answer, only a mismatch between engine and load.

Whichever you choose, plan for measurement. Time to first token, tokens per second, and queue depth under real concurrency tell you more than any published benchmark, and they should be tracked with self-hosted LLM observability rather than an external service, since prompts and retrieved passages carry the same regulated data you deployed on-premise to protect.

Where does the inference server fit in a private RAG system?

The inference server is one of four layers in a private RAG deployment: serving, retrieval (embeddings plus a vector database), orchestration, and governance (permissions and audit). Choosing vLLM, Ollama, or llama.cpp settles how fast and how widely the model can serve, not whether answers are grounded, permission-filtered, or traceable. Those properties come from the layers above the serving tier.

Samvad AI is Teclops AI’s secure, source-cited RAG assistant that answers only from your own documents and packages those layers so your team is not assembling and tuning a serving stack by hand. It deploys on-premise, air-gapped, or hybrid (switchable by configuration), enforces role- and row-level permissions, cites the exact source passage behind every answer, says plainly when an answer is not in your sources, and keeps a tamper-evident audit log. If you would rather have the serving, retrieval, and governance layers designed and built inside your own infrastructure, Teclops AI’s AI product development and automation services cover exactly that. Reach the team at teclops.ai@gmail.com.