Small vs Large LLMs: What Size Model Do You Need?

Most enterprise AI workloads do not need a frontier-scale model: extraction, classification, summarization, and source-cited question-answering over your own documents run well on open-weight models in the 8B to 32B range. The reason is that retrieval quality, not parameter count, decides accuracy on internal-knowledge questions. Large models earn their cost on a narrower set of tasks where the answer has to be composed rather than located. This guide gives you a sizing ladder, a table by use case, and the honest list of places where a small model breaks.

What is the difference between a small and a large language model?

A small language model (SLM) is a language model of roughly 1 to 10 billion parameters, small enough to serve from a single GPU. The boundary is a convention rather than a standard, and different vendors draw it anywhere from 7B to the low teens. A large language model in the enterprise sense is a 70-billion-parameter dense model, or a mixture-of-experts model with hundreds of billions of total parameters, needing multi-GPU memory to serve. Between them sits the mid-size class, roughly 12B to 40B, which is where a large share of production enterprise work actually lands.

The practical dividing line on-premise is not marketing language, it is VRAM. Weights consume about 0.5 GB per billion parameters at 4-bit and 1 GB at 8-bit, plus key-value cache. A model that fits one GPU is an operational decision; a model that needs a multi-GPU node is a procurement decision. For the full math, see air-gapped LLM hardware and GPU VRAM sizing.

Does model size or retrieval quality matter more for RAG?

For retrieval-augmented generation, retrieval quality matters more than model size. If the correct passage is in the context window, reading it, quoting it, and citing it is a comparatively easy task that mid-size and even small models perform reliably. If the correct passage is missing, no amount of parameters recovers it, because a general-purpose model has never seen your loan files, clinical protocols, or procurement contracts.

This is why the common upgrade instinct is usually wrong. When a grounded assistant gives a bad answer, the fix is far more often better chunking, hybrid search, or a reranker than a bigger model. Diagnose before you spend: measure retrieval hit rate separately from answer faithfulness, using the method in how to evaluate RAG accuracy. If retrieval only finds the right passage seven times in ten, a larger model cannot lift the system past that ceiling.

How do you decide what size model a task needs?

Use the locate, compose, invent ladder. It is the simplest reliable predictor of how much model a task needs, because it describes how much of the answer the model must produce from its own weights.

1. Locate. The answer exists verbatim in a retrieved passage. The model finds it, restates it, and cites it. Small models handle this well.
2. Compose. The answer exists across several passages and must be assembled, compared, or reconciled. This is mid-size territory, and multi-hop questions push toward the top of it.
3. Invent. The answer is not in any source. The model must reason, plan, or draft from general capability. This is where large models earn their cost.

The strategic point follows directly: the job of a good RAG system is to move tasks down the ladder, from invent to locate. Every improvement in retrieval converts an inventing task into a locating task, and locating tasks need far less model. Teams who skip retrieval work end up buying parameters to compensate for context they could have supplied cheaply.

Four questions then set the size within a rung:

• How many reasoning hops? One hop is small-model work. Three or more hops across documents is not.
• How strict is the output contract? Long JSON schemas, nested conditionals, and rigid citation formats stress small models before content does.
• Which languages? Low-resource language nuance is one of the clearest genuine gains from scale.
• How adversarial is the input? Untrusted document content raises the bar on instruction-following and refusal discipline.

What size model do you need for each use case?

The table below maps common enterprise workloads to a starting size class. Treat the starting class as the first model you test, not the answer, and size up only when your own evaluation shows a real gap.

Use case	Start here	Why that size is usually enough	Size up when
Field extraction from a retrieved document (dates, amounts, parties)	3B to 8B	The value is present in the text; the task is locating and copying	Schema is long, scans are noisy, rules are conditional
Classification, routing, triage	1B to 8B	Narrow label space, short output	Many labels with subtle boundaries
Single-document summarization	7B to 14B	Compression, not inference	Faithful summaries of long legal or clinical text
Source-cited Q&A over your own documents	8B to 32B	Retrieval supplies the facts; the model reads and cites	Questions span many documents
Natural-language analytics over a governed semantic layer	8B to 32B	The semantic layer constrains the space of valid answers	Ambiguous questions, heavy multi-metric reasoning
Multi-document synthesis, comparison, contradiction-finding	30B to 70B	Several sources must be held and reasoned across at once	Corpus is large and conclusions carry risk
Agentic workflows with tools and multi-step plans	30B to 70B+	Planning, tool choice, and error recovery compound per step	Steps are long or actions are irreversible
Open-ended drafting or reasoning with no source to ground against	70B+	Nothing to retrieve; output comes from the weights	Quality is the product, not the input

How does model size affect latency and cost on owned GPUs?

On hardware you own, model size drives three things at once: how fast a single answer streams, how many users you can serve concurrently, and what your effective cost per answer becomes. Token generation is largely memory-bandwidth-bound, so decode speed scales roughly inversely with the bytes of weights read per token. A much smaller model is therefore substantially faster on the same GPU, which matters most for interactive assistants where users judge quality partly by responsiveness.

Concurrency is the underrated half. Smaller weights leave more VRAM for key-value cache, and cache is what limits simultaneous requests. On-premise cost is mostly fixed (amortized GPU capital plus power and operations), so cost per answer falls as the same box serves more people. A smaller model raises throughput on hardware you have already bought, which is a cheaper lever than buying more GPUs.

There is a second way to spend the savings well: put the freed VRAM and latency budget into retrieval. A reranker pass, a larger retrieved context, and a verification step on a small model often beat a bare large model on grounded accuracy, and they cost less to serve.

Where do small language models genuinely fail?

Small language models genuinely fail on tasks that require holding and manipulating many things at once. Being specific about this matters, because the failure modes are predictable:

• Multi-hop reasoning. Answers that require chaining three or more facts across documents degrade before single-hop answers do.
• Instruction stacking. Grounding rules, citation format, tone, refusal policy, and a long output schema applied simultaneously is where small models drop a constraint.
• Long-context discipline. With many retrieved passages, smaller models are more likely to fixate on one passage or blend two sources into a single wrong claim.
• Refusal calibration. Saying “this is not in the sources” is a learned behavior, and larger models generally hold that line more consistently under pressure.
• Low-resource language nuance. Coverage and tokenization quality vary sharply by model family, and this is a real reason to size up or to choose a strongly multilingual family.
• Adversarial robustness. Untrusted content inside retrieved documents is harder for a small model to resist, which is why prompt injection defenses in enterprise RAG belong in the architecture, not in the model.

Most of these are composition and instruction failures, not knowledge failures. That is consistent with the ladder: small models are weak at inventing and adequate at locating.

Should you run more than one model size?

Running two model sizes behind one interface is usually better economics than standardizing on one. The pattern is tiered routing: send extraction, classification, and simple grounded lookups to a small model, and reserve a larger model for multi-hop synthesis, agentic steps, and anything a confidence or complexity check flags. Most enterprise query volume is simple, so most traffic never touches the expensive path.

Two conditions make tiered routing work. First, routing must be observable, so you can see which tier answered and audit it later. Second, the quality bar must be measured per tier on your own test set, not assumed. Gist, Teclops AI’s agentic analytics product, keeps governed, no-SQL natural-language analytics affordable for smaller teams without giving up answer quality on hard questions.

How does Teclops AI size models for on-premise deployments?

Samvad AI is Teclops AI’s secure, source-cited RAG assistant, and it is built on the principle above: it answers only from your own documents, cites the exact source passage, and says plainly when an answer is not in your sources. That grounding is what lets a right-sized model outperform a much larger ungrounded one. Samvad AI deploys on-premise, air-gapped, or hybrid (switchable by configuration), is permission-aware at the role and row level, is multilingual, and keeps a tamper-evident audit log.

In practice, the method is simple: start with the smallest model that clears your accuracy bar on your own documents and questions, invest the savings in retrieval quality, and size up only where evaluation proves a gap. If you want the sizing, evaluation, and serving handled inside your own perimeter, reach the team at teclops.ai@gmail.com.