What Is LLM Quantization? GGUF, AWQ, GPTQ, FP8
LLM quantization stores model weights at lower precision to cut GPU memory: 4-bit saves roughly 65 to 70 percent versus FP16, 8-bit about 50 percent.
LLM quantization is the practice of storing a model’s weights at lower numeric precision than the 16-bit format it was trained in, cutting weight memory roughly in half at 8-bit and by about 65 to 70 percent at 4-bit. That compression is what lets a 70B open-weight model run on a single GPU inside your own data centre. It is also lossy, and the loss is not spread evenly across tasks. This guide compares GGUF, GPTQ, AWQ, FP8, and INT8, maps each format to the inference server that runs it, and sets out when quantization is safe for enterprise RAG and when it quietly degrades answers.
What is LLM quantization?
LLM quantization is a compression technique that represents each model parameter with fewer bits. A model trained in 16-bit precision uses 2 bytes per parameter; quantizing to 8-bit uses about 1 byte, and to 4-bit about 0.5 bytes for the quantized layers, plus overhead for the scaling factors that map compressed values back to a usable range and for the layers a format leaves at higher precision (embeddings and the output head are commonly kept larger). Weights are handled in groups (commonly 64 or 128 values), and each group gets its own scale, so the compression is lossy but locally calibrated.
Two distinctions decide most on-premise deployments:
- Post-training quantization (PTQ) compresses an already-trained model using a small calibration dataset. GPTQ, AWQ, FP8, and INT8 checkpoints are all PTQ, and this is what almost every enterprise uses, because it needs no training run.
- Weight-only versus weight-and-activation. GGUF, GPTQ, and AWQ compress weights only and compute in higher precision after dequantizing. FP8 and INT8 W8A8 compress weights and activations, which is what enables faster math on the GPU rather than only a smaller memory footprint.
What do 4-bit and 8-bit mean for VRAM?
Bit width translates directly into GPU memory: a model needs roughly 2 GB of VRAM per billion parameters at 16-bit, 1 GB at 8-bit, and around 0.6 GB at 4-bit once the unquantized layers are counted, plus headroom on top for the key-value cache and activations.
| Precision | Bytes per parameter | 8B model weights | 70B model weights |
|---|---|---|---|
| FP16 / BF16 | ~2.0 | ~16 GB | ~140 GB |
| FP8 / INT8 | ~1.0 | ~8 GB | ~70 GB |
| 4-bit (AWQ, GPTQ, Q4_K_M) | ~0.6, higher on small models | ~5 to 6 GB | ~40 to 43 GB |
One caveat gets missed constantly: quantization shrinks the weights, not the KV cache. The cache scales with concurrent users and context length and stays at its original precision unless KV-cache quantization is enabled separately. Size the server for both. The full sizing method, including CPU, RAM, and disk, is in the guide to air-gapped LLM hardware and GPU VRAM sizing.
GGUF vs GPTQ vs AWQ vs FP8: which format should you use?
These four are not competitors on a single axis. GGUF is a file format built for the llama.cpp runtime; GPTQ and AWQ are 4-bit weight-only methods for GPU serving; FP8 is a hardware-accelerated 8-bit number format. The table compares them on the terms that decide an on-premise deployment.
| Format | Typical bit width | What it compresses | Primary runtime | Weight VRAM vs FP16 | Best fit |
|---|---|---|---|---|---|
| GGUF (Q4_K_M, Q5_K_M, Q8_0) | about 1.5 to 8-bit, mixed per layer | Weights | llama.cpp, Ollama | Q4_K_M ~70% smaller; Q8_0 ~50% | CPU-only, mixed CPU and GPU offload, single-user or small-team boxes, edge sites |
| GPTQ | 4-bit (also 2, 3, and 8) | Weights only | vLLM, Transformers | ~65 to 70% smaller | 4-bit serving on older GPU generations, wide checkpoint availability |
| AWQ | 4-bit | Weights only | vLLM, Transformers | ~65 to 70% smaller | Strong-quality 4-bit weight-only for GPU serving; fitting a bigger model on fewer GPUs |
| FP8 (E4M3) | 8-bit float | Weights, activations, optional KV cache | vLLM, TensorRT-LLM, SGLang | ~50% smaller | High-throughput multi-user serving on Hopper, Ada, or Blackwell class GPUs |
| INT8 W8A8 | 8-bit integer | Weights and activations | vLLM | ~50% smaller | 8-bit throughput on GPUs without native FP8 support |
| BF16 / FP16 | 16-bit | Nothing | Any | Baseline | Accuracy reference for evaluation, and small models where VRAM is not the constraint |
A decision rule that covers most on-premise cases:
- Multi-user GPU serving on Hopper class or newer: start at FP8. It carries the least quality risk and holds up best under load.
- The model does not fit, or the GPUs are older: use AWQ 4-bit, falling back to GPTQ if no AWQ checkpoint exists for that model.
- Older GPUs, 8-bit throughput wanted: use INT8 W8A8.
- One machine, few users, CPU or partial GPU offload: use GGUF at Q4_K_M or Q5_K_M, or Q8_0 if the memory is there.
- Accuracy-critical work: keep a BF16 copy as the control to measure everything else against.
Which quantization format pairs with which inference server?
The runtime decides the format, so choose the serving stack before the checkpoint. llama.cpp and Ollama consume GGUF and are built for flexible CPU and GPU-offload execution on a single box, which suits a branch office, a lab machine, or an edge site with one consumer GPU. vLLM, TensorRT-LLM, and SGLang are throughput-oriented GPU servers with continuous batching and paged attention, and they consume AWQ, GPTQ, FP8, and INT8 checkpoints rather than GGUF. Converting a GGUF file for a high-throughput GPU server is usually the wrong move: obtain a checkpoint already in the format that server expects. For how the inference server sits alongside retrieval and governance, see the architecture walkthrough on deploying an LLM on-premise.
Does quantization make inference faster or slower?
Quantization moves speed in two opposite directions depending on load, which is why single-user benchmarks mislead teams sizing a shared enterprise server.
- At low batch sizes, token generation is memory-bandwidth bound: the GPU spends its time moving weights, so 4-bit weight-only formats such as AWQ and GPTQ generate noticeably faster than FP16 because there is less to move.
- At high batch sizes, the workload becomes compute bound, and weight-only 4-bit formats pay a dequantization cost on every forward pass. FP8 and INT8 W8A8 can then be faster than 4-bit, because the matrix math itself runs at low precision in hardware.
If your deployment serves many concurrent users, do not assume the smallest format is the fastest. Test at your real concurrency.
When is quantization safe for enterprise RAG, and when does it degrade accuracy?
Quantization is comparatively safe for retrieval-augmented generation, because the model is mostly reading and summarizing text placed in its context window rather than recalling facts from its weights. Extraction, grounded question answering, and passage summarization tolerate 4-bit better than open-ended knowledge tasks do, and 8-bit formats are commonly close to indistinguishable from BF16 on these workloads. Treat that as a starting expectation, not a guarantee.
Quantization damage concentrates in six places, and these are the ones to test hard:
- Small models. A 3B or 8B model has less redundancy than a 70B one, so the same 4-bit compression costs proportionally more quality.
- Multi-step reasoning and arithmetic. Errors compound across steps, and numeric and tabular reasoning is sensitive to small precision losses.
- Low-resource and non-English languages. Calibration data is usually English-heavy, so Hindi, Tamil, and other languages can degrade more than an English benchmark reveals.
- Strict structured output. JSON schemas, tool calls, and citation markers break in ways that fluent prose hides.
- Refusal behaviour. A quantized model may become more willing to guess when the answer is not in the retrieved passages, which is the worst failure mode in a regulated deployment.
- Very long context. Quality loss tends to widen as context grows, and long retrieved contexts are normal in RAG.
How do you validate a quantized model before it serves users?
Validate a quantized model the way you would validate any model change: against a golden test set of your own questions and source passages, with the BF16 version as the control. Score both on faithfulness, citation correctness, and refusal-when-unknown rather than fluency, and compare deltas per category instead of one headline number. The method is set out in how to evaluate RAG accuracy.
Three habits keep this honest in a regulated environment:
- Record the exact quantization in your change log, including method, bit width, group size, and calibration set, alongside model name and version. A bare model name is not a sufficient description of what you deployed.
- Include non-English and long-context questions in the test set, because those are where quantization damage hides.
- Re-run the evaluation on every format swap, and treat a quantization change as a controlled change with a rollback path, exactly as you would a model upgrade.
How does Samvad AI handle quantization?
Samvad AI is Teclops AI’s secure, source-cited RAG assistant that runs open-weight models on your own hardware, so the quantization format, the serving stack, and the validation that follows a model swap are part of the deployment rather than your team’s problem. Samvad AI answers only from your own documents, cites the exact source passage, says plainly when an answer is not in your sources, and deploys on-premise, air-gapped, or hybrid, switchable by configuration. Because it is permission-aware and keeps a tamper-evident audit log, the record of which model and which quantization produced an answer stays inside your perimeter. To have the format chosen, benchmarked on your documents, and served on hardware you own, reach the team at teclops.ai@gmail.com.
Frequently asked questions
Which quantization format is best for enterprise RAG?
On current data-center GPUs with native FP8 support, FP8 is the safest default for enterprise RAG because it halves weight memory with very little quality change under multi-user load. Where the model will not fit or the GPUs are older, AWQ 4-bit is the usual pick. Either way, score the candidate against a golden test set of your own documents before it serves users.
Is AWQ better than GPTQ?
AWQ and GPTQ are both 4-bit weight-only post-training methods that differ in approach: AWQ protects the small fraction of weight channels that activations show to be most important, while GPTQ minimises layer-wise reconstruction error using approximate second-order information. AWQ generally holds accuracy at least as well at the same bit width and has strong serving kernels, so it is the more common GPU default, with GPTQ useful when no AWQ checkpoint exists for a model.
Can you run GGUF models in vLLM?
GGUF support in vLLM exists but is experimental and limited, because GGUF was designed for the llama.cpp runtime that also powers Ollama. For throughput-oriented GPU serving, use an AWQ, GPTQ, FP8, or INT8 checkpoint rather than converting a GGUF file, and keep GGUF for CPU, mixed CPU and GPU offload, or single-user local deployments.
What is the difference between FP8 and INT8 quantization?
FP8 is an 8-bit floating-point format (usually E4M3) with an exponent field, so it covers a wide dynamic range and is accelerated in hardware on Hopper, Ada, and Blackwell class GPUs. INT8 is an 8-bit integer format that needs calibration techniques such as activation smoothing to handle outlier values, but it runs on older GPU generations that lack native FP8 tensor cores.
Does quantizing a model also shrink the KV cache?
No. Quantizing weights to 4-bit or 8-bit reduces only the memory the weights occupy. The key-value cache, which grows with every concurrent request and every token of context, stays at its original precision unless KV-cache quantization is enabled separately, so servers sized on quantized weight size alone often run out of VRAM under real multi-user load.