LoRA vs Full Fine-Tuning for On-Prem LLMs

LoRA fine-tunes an open-weight LLM by freezing the original weights and training small adapter matrices alongside them, typically under 1 percent of the parameters, which fits on a single GPU. Full fine-tuning updates every weight and needs roughly 16 GB of GPU memory per billion parameters, meaning a multi-GPU node even for an 8B model. For nearly every on-premise enterprise use case, LoRA or QLoRA is the correct default. Full fine-tuning is reserved for genuine domain adaptation at scale.

This guide assumes you have already ruled out retrieval. If you have not, start with RAG vs fine-tuning for enterprise, because fine-tuning teaches behavior rather than facts, and a fine-tuned model cannot cite a source.

What is LoRA, and how does it differ from full fine-tuning?

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a base model’s weights and injects small trainable rank-decomposition matrices into selected layers, usually the attention projections and sometimes the MLP blocks. Only those adapter matrices receive gradients, so the base model is never modified and its weights remain byte-identical on disk.

Full fine-tuning updates all of a model’s parameters. Every weight receives a gradient, which is why the memory cost is dominated not by the weights themselves but by the gradients and optimizer states that accompany them.

Three practical consequences follow from that difference:

• Memory. A LoRA run stores optimizer state for a tiny parameter set. A full fine-tune stores a gradient and two Adam moments for every parameter.
• Reversibility. A LoRA adapter is a small file you can unload, version, or swap. A full fine-tune produces a new multi-gigabyte checkpoint, and rollback means restoring the previous one.
• Serving. One base model in VRAM can serve several LoRA adapters, which matters when GPUs are the scarce resource on-premise. Merging an adapter into the base weights removes the small inference overhead when you need it.

When is fine-tuning actually warranted?

Fine-tuning is warranted only when the gap between what your model does and what you need is behavioral, repeatable, and unfixable by retrieval or prompting. Before provisioning a single GPU-hour, run three gates. If any answer is no, fine-tuning is the wrong tool.

1. Is the gap behavior, not knowledge? “The model does not know our Q3 policy” is a retrieval problem, and fine-tuning makes it worse by baking a stale fact into weights with no provenance. “The model will not reliably emit our claim-adjudication schema” is a behavior problem, and fine-tuning is a legitimate fix.
2. Has prompting genuinely failed? A structured system prompt plus few-shot examples solves a large share of format and tone problems at zero training cost and zero rollback risk. Fine-tune after a measured prompting attempt fails, not instead of one.
3. Do you have graded examples and a way to measure the result? You need a few hundred correct input-output pairs and a held-out test set you did not train on. Without measurement you cannot distinguish an improvement from a regression.

The most common expensive mistake in enterprise AI is fine-tuning to solve a retrieval problem. The second is fine-tuning without a held-out evaluation set.

LoRA vs full fine-tuning: side-by-side comparison

Dimension	LoRA / QLoRA	Full fine-tuning
What changes	Small injected adapter matrices; base weights frozen	Every parameter in the model
Trainable parameters	Typically well under 1 percent of the model	100 percent
GPU memory rule of thumb	Base weights only (about 2 GB per billion at bf16, 0.5 GB at 4-bit) plus activations	About 16 GB per billion for weights, gradients, and Adam states, plus activations
8B model, practical rig	One 24 GB GPU (QLoRA) or one 48 GB GPU (bf16)	Multi-GPU node, roughly 128 GB and up with sharding
70B model, practical rig	One 48 to 80 GB GPU with QLoRA	A multi-node cluster
Training data needed	Hundreds to a few thousand curated examples	Substantially more; its advantage grows with corpus size
Catastrophic forgetting	Contained; base capability preserved, adapter can be disabled	Real risk; general ability can degrade while task accuracy rises
Rollback	Unload or version the adapter file	Restore a full previous checkpoint
Serving	One base model can host multiple swappable adapters	One dedicated model per tuned behavior
Iteration speed	Hours; cheap to rerun with new data	Days; each rerun is a budget decision
Best for	Format, tone, narrow tasks, per-department behavior	Deep domain or language adaptation with a large corpus
Main limitation	Ceiling on how far behavior can move from the base	Cost, hardware, and regression risk

What is QLoRA, and when do you need it?

QLoRA is LoRA applied on top of a base model quantized to 4-bit, with the adapters trained in higher precision. It cuts the memory needed to hold the frozen base by roughly four times versus bf16, which is what makes fine-tuning large models on modest hardware possible at all. The authors of the original QLoRA work reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU.

Use QLoRA when the frozen base does not fit your GPU at 16-bit: at roughly 2 GB per billion parameters, that is anything above about 7B on a 24 GB card, or above about 13B on a 48 GB card once activations are accounted for. Use plain LoRA at bf16 when the base fits comfortably, since you avoid quantization error during training and a somewhat slower step time. Either way, evaluate the quantized-plus-adapter configuration you intend to serve, not a higher-precision approximation of it.

How much GPU memory does on-premise fine-tuning really need?

Fine-tuning memory is a different calculation from inference memory. At inference you pay for weights plus the key-value cache. During training you also pay for gradients, optimizer states, and stored activations for the backward pass.

• Full fine-tuning with Adam in mixed precision: roughly 16 bytes per parameter (an fp32 master weight, a gradient, two optimizer moments), so about 16 GB per billion parameters, plus activations that scale with sequence length and batch size. An 8B model needs on the order of 128 GB before activations. Sharding spreads that across GPUs; it does not reduce the total you must own.
• LoRA at bf16: about 2 GB per billion for the frozen base, plus small adapter optimizer state and activations. An 8B model is comfortable on one 48 GB GPU.
• QLoRA at 4-bit: about 0.5 GB per billion for the frozen base. An 8B model needs roughly 5 GB of weights and trains on a 24 GB card, and a 70B model fits a single 48 to 80 GB GPU.

That gap, one workstation-class GPU versus a cluster, is why LoRA dominates on-premise practice. For the underlying bytes-per-parameter math and model-to-GPU mapping, see air-gapped LLM hardware and GPU VRAM sizing.

How do you prepare a fine-tuning dataset inside your own perimeter?

Dataset quality determines a fine-tuning outcome more than any hyperparameter, so build the corpus with production-pipeline controls on infrastructure you own.

1. Define the target behavior in one sentence. “Always return the four-field risk summary in this exact JSON shape” is a target. “Be better at compliance” is not.
2. Collect real examples, not invented ones. Pull from actual tickets, reports, and expert-written outputs. Synthetic examples teach the model to imitate a generator, including its errors.
3. Have an expert grade every output. A few hundred verified-correct pairs beat tens of thousands of scraped ones, because the model faithfully learns whatever inconsistency you leave in the labels.
4. Normalize and deduplicate. One prompt template, one output shape, consistent field order. Near-duplicates inflate evaluation scores, and any example appearing in both training and test makes results meaningless.
5. Minimize sensitive data. Redact or pseudonymize identifiers the behavior does not require. Whatever you train on is absorbed into the weights, and models are known to memorize and sometimes reproduce fragments of training data.
6. Hold out a test split before training, and record provenance. Split by document, customer, or time period rather than randomly by row. Log which source systems contributed, under what authority, and when the dataset expires, because the manifest is your answer when auditors ask what is inside the model.

How do you evaluate a fine-tuned model before rollout?

Evaluate a fine-tuned model on two axes: did it learn the target behavior, and did it lose anything else. Measuring only the first is how teams ship a model that formats beautifully and reasons worse than the one it replaced.

1. Score the target task on the held-out split, using exact-match on schema, format compliance rate, or classification accuracy.
2. Run a general-capability regression suite. Instruction following, multi-step reasoning, the other languages your users need, and refusal behavior on out-of-scope questions, compared against the base model on identical prompts.
3. Test the retrieval path if the model serves RAG. A tuned model that ignores retrieved context or stops citing correctly is a failure however good its formatting is.
4. Probe for training-data leakage, particularly if any sensitive text survived redaction.
5. Compare against the honest baseline. The baseline is not “no model.” It is the base model with a good prompt and retrieval. If the fine-tune does not clearly beat that, do not ship it.

What is catastrophic forgetting, and how do you limit it?

Catastrophic forgetting is the degradation of a model’s existing capabilities as it is trained on a narrow new task: the model improves at what you showed it and gets measurably worse at instruction following, other languages, reasoning, and refusal behavior. It is the most under-tested failure mode in enterprise fine-tuning, because training loss looks excellent while general quality quietly falls.

Full fine-tuning carries the higher risk, since every weight moves. LoRA contains the risk structurally: base weights are frozen, so original capability remains, and the adapter can be unloaded or its influence scaled down. Containment is not immunity. A high-rank adapter trained at an aggressive learning rate over many epochs can still push behavior far enough to break general performance. The mitigations are conservative: start with a low rank, use a smaller learning rate than feels necessary, train for few epochs, and keep the checkpoint where held-out general capability is still intact rather than the one with the lowest training loss.

How do you keep training data and model weights inside the perimeter?

On-premise fine-tuning keeps the corpus, the checkpoints, and the training logs on hardware you control, which is the entire point for a bank, hospital, university, or government body. A hosted fine-tuning API inverts that: your most sensitive labeled data is transmitted to a third party, and the resulting weights live under someone else’s access controls. Four controls matter in practice.

• Stage dependencies offline. Base weights, training framework, CUDA libraries, and Python packages come in through your normal offline transfer and integrity-check process. Nothing is pulled from a public registry mid-run.
• Disable telemetry and experiment-tracking defaults. Several popular training libraries call out to hosted tracking services unless configured otherwise. Point them at a self-hosted endpoint or turn them off, and verify at the network layer rather than trusting the config.
• Treat weights as a sensitive asset. A fine-tuned checkpoint or adapter encodes your training data. Store it under the same classification as the source corpus, control who can copy it, and log access.
• Keep permissions where permissions work. You cannot apply per-user access control to knowledge that now lives inside the weights. Anything that must respect role- or row-level permissions belongs in a retrieval store queried under access checks at answer time. That is why Samvad AI, our source-cited on-premise RAG assistant, grounds answers in retrieved documents and cites the exact source passage instead of memorizing documents into a model.

Should you choose LoRA, QLoRA, full fine-tuning, or none of them?

• Choose LoRA for a consistent output schema, a domain tone, or a narrow classification task, when you have hundreds to a few thousand graded examples and one or two GPUs. This covers the large majority of enterprise cases.
• Choose QLoRA when the base model does not fit your GPU at 16-bit, for example tuning a 70B model on a single 48 GB card, and evaluate the quantized configuration you will actually serve.
• Choose full fine-tuning only for genuine domain or language adaptation with a large corpus, a measured gap that LoRA failed to close, and cluster-scale hardware plus the budget to rerun it.
• Choose none of them when the gap is factual, changes as documents change, or must respect per-user permissions. That is a retrieval problem, and no amount of training fixes it.

Teclops AI builds and evaluates these systems inside the client’s own infrastructure, from dataset preparation and adapter training through evaluation and offline deployment. If you are still deciding whether a fine-tune is warranted at all, our AI product development and consultancy services start with that question rather than assuming the answer. Reach the team at teclops.ai@gmail.com.