Air-Gapped LLM Hardware: How Much VRAM?

Air-gapped LLM hardware is sized primarily by GPU memory (VRAM), and the rule of thumb is about 0.5 GB of VRAM per billion parameters at 4-bit quantization, 1 GB at 8-bit, and 2 GB at 16-bit, plus 20 to 40 percent extra for the key-value cache. An 8B open-weight model fits comfortably on a single 24 GB GPU; a 70B model at 4-bit needs roughly 40 to 48 GB. This guide gives you the sizing math, a model-to-GPU table for currently shipping open-weight models, and the extra headroom an offline deployment adds on top.

What hardware do you need to run an LLM fully offline?

Running an LLM fully offline (air-gapped, with no internet, no telemetry, and weights stored locally) requires four things on hardware you own: one or more GPUs with enough total VRAM to hold the model weights plus the key-value cache, system RAM at least equal to the model’s on-disk size, fast local disk for the weight files, and CPU for retrieval and orchestration. GPU VRAM is the binding constraint; everything else is comparatively cheap. Because an air-gapped system pulls no live updates and calls no hosted API, you size for the peak load the box will ever serve, since you cannot burst to the cloud when demand spikes.

How do you calculate VRAM for an open-weight model?

You calculate VRAM for an open-weight model by multiplying the parameter count (in billions) by the bytes per parameter for your quantization, then adding overhead for the cache and activations. The bytes per parameter are fixed:

• 16-bit (FP16 / BF16): 2 bytes per parameter
• 8-bit (INT8): about 1 byte per parameter
• 4-bit (INT4): about 0.5 bytes per parameter

A 32B model at 4-bit needs roughly 32 x 0.5 = 16 GB for weights, about 20 GB once you add cache headroom. A 70B model at 8-bit needs about 70 GB for weights alone, which is why it does not fit a single 80 GB GPU once concurrency is added. Add 20 to 40 percent on top of the weight figure for the key-value cache, and more if you serve long contexts or many simultaneous users.

Which GPU and VRAM does each open-weight model need?

The table below maps common, currently shipping open-weight model sizes to their approximate VRAM footprint at each quantization and the smallest practical GPU tier at 4-bit. Figures are weights plus modest cache headroom; treat them as planning estimates, not guarantees.

A note on mixture-of-experts (MoE) models: the total parameter count is what must fit in VRAM, even though only a subset of experts is active per token. Llama 4 Scout is the clearest example, with 17B active parameters but 109B total, so you size VRAM for 109B. Gemma 4’s 26B-A4B is 25.2B total with 3.8B active. MoE reduces compute and improves speed, not the memory you must own. The parameter counts here are the published figures; the VRAM numbers follow from the bytes-per-parameter math above.

How much VRAM headroom do you need for concurrency and context?

You need meaningful VRAM headroom beyond the model weights, because every concurrent request consumes additional memory for its key-value cache, and that cache grows with context length. A single user at short context might add only a few hundred megabytes; ten simultaneous users at long context can add many gigabytes. This is the most common sizing mistake in air-gapped deployments: teams buy exactly enough VRAM for the weights, then find the system stalls or refuses requests under real multi-user load.

Practical guidance:

1. Size the weights from the table, then reserve 20 to 40 percent on top for cache and activations at expected concurrency.
2. If long context (large retrieved passages plus the question) is normal, budget toward the high end.
3. Leave room to load a second model version side by side, so you can validate and roll back an offline model swap without downtime.

What system RAM, CPU, and disk does an air-gapped LLM need?

Beyond the GPU, an air-gapped LLM node needs enough system RAM, CPU, and disk to load weights and run retrieval. System RAM should be at least the model’s on-disk size and ideally 1.5 to 2 times total VRAM, so weights stage cleanly and the operating system, embedding model, and vector database have room. Disk must hold every weight file you keep on hand, including multiple quantizations and prior versions, so plan for several times a single model’s size. The retrieval layer (embedding model plus vector database) is light compared with the LLM and often runs on CPU or shares the same host, a split covered in the guide to deploying an LLM on-premise.

How does air-gapped sizing differ from cloud sizing?

Air-gapped sizing differs from cloud sizing because you cannot elastically scale, you cannot offload to a managed endpoint, and you must provision for peak load permanently. In the cloud you rent capacity by the hour and let the provider absorb bursts; air-gapped, the hardware you buy is the ceiling. Three consequences follow:

• Provision for peak, not average. There is no burst path, so the box must handle your busiest hour.
• Quantization is a first-class lever. Dropping from 8-bit to 4-bit roughly halves VRAM, often letting a capable model fit a single GPU with acceptable quality loss.
• Updates are files, not API calls. Model swaps arrive as signed weight files through controlled transfer, so disk and staging capacity matter more than bandwidth.

For the broader control and cost tradeoff behind this choice, see on-prem vs cloud LLM.

Which quantization should you choose for offline deployment?

Choose the lowest quantization that still clears your accuracy bar on your own data, because quantization is the cheapest way to fit a stronger model on the hardware you have. For most retrieval and question-answering workloads, teams commonly report that 4-bit costs little accuracy while roughly halving VRAM versus 8-bit, which is what lets a 70B model run on a single GPU air-gapped. Treat that as a starting expectation rather than a guarantee: the loss varies by quantization method, model family, and task, and it tends to be larger on smaller models and reasoning-heavy work. Reserve 8-bit or 16-bit for tasks where you have measured a real quality gap. Always validate the quantized model on your own documents and questions rather than trusting a general benchmark, and pair that with the license and task-fit checks in how to choose an open-weight LLM for on-premise use.

Who specs and serves the air-gapped stack for you?

Samvad AI is Teclops AI’s secure, source-cited RAG assistant that runs open-weight models on your hardware and answers only from your own documents, so the sizing, serving, and offline model lifecycle are handled inside your perimeter rather than left to your team. Samvad AI deploys on-premise, air-gapped, or hybrid (switchable by configuration), cites the exact source passage for every answer, and keeps a tamper-evident audit log. For governed natural-language analytics on the same principle, Gist runs inside your infrastructure with row- and column-level security. If you want the hardware specced and the stack served within your own network, reach the team at teclops.ai@gmail.com.