How to Choose an Embedding Model for On-Premise RAG
Choose an on-premise embedding model by license first, then max sequence length, language coverage, and domain fit, weighing dimensions against storage cost.
Choose an embedding model for on-premise RAG by hard constraints first: a license that permits commercial self-hosting, weights that run fully offline, a sequence length that fits your chunks, and real coverage of your users’ languages. Accuracy comparisons come after those filters, and they should be run on your own documents rather than a public leaderboard. The choice deserves more care than picking the generation model, for one structural reason: your LLM is swappable at runtime, and your embedding model is not.
What is an embedding model in a RAG system?
An embedding model is a neural network that converts a piece of text into a fixed-length numeric vector, so that passages with similar meaning sit close together in vector space and a user’s question can be matched to relevant chunks by distance rather than by keyword. In a RAG pipeline the same embedding model runs twice: once over every document chunk at ingestion time, and again over each incoming query at retrieval time. Both sides must use the same model and the same version, because the comparison is only meaningful within one vector space.
Why does the embedding model matter more than the generation model?
The embedding model is the schema of your vector index, not a runtime setting. Every chunk in your corpus is stored as a vector produced by one specific model at one specific version, and vectors from two different models are not comparable, so changing the embedding model invalidates the entire index. You can swap a generation model on a Tuesday afternoon and swap it back on Wednesday. Changing the embedding model means re-embedding every document, rebuilding the index, re-tuning retrieval thresholds, and re-validating accuracy against your test set.
That asymmetry should drive the selection process. Treat the generation model as a reversible choice and the embedding model as a storage commitment with a migration cost attached.
What should you evaluate first when selecting an embedding model?
Evaluate embedding models as an ordered filter, so you only spend evaluation time on candidates that are actually deployable inside your perimeter:
- License. Confirm the weights permit commercial use and self-hosting. Some strong multilingual embedding models ship under non-commercial terms, which disqualifies them for a bank or an insurer regardless of accuracy.
- Offline operability. The weights must run with no outbound call, no license server, and no telemetry. Check whether the model requires custom modelling code (
trust_remote_code), because that code needs a security review before it runs inside a regulated network. - Max sequence length. A model that truncates at 512 tokens will silently discard the tail of a 900-token chunk. Match the limit to your chunking strategy, or change your chunking.
- Language coverage. Verify the specific languages your users type in, not a marketing count of “100+ languages”.
- Dimensions and footprint. Decide what you are willing to store and serve, using the storage table below.
- Domain fit on your data. Only now run the two or three survivors against a labelled test set of your own questions.
Which criteria decide an on-premise embedding model choice?
The table below is a scorecard for selecting an embedding model that will run inside your own infrastructure. Each row states what good looks like and the red flag that should stop the evaluation.
| Criterion | What good looks like | The red flag |
|---|---|---|
| License | Permissive terms (such as Apache 2.0 or MIT) on the exact weights you download | Non-commercial terms, or a license that differs by model size |
| Offline operation | Static weight files, runs air-gapped, no phone-home | Remote code execution required, or hosted-only inference |
| Max sequence length | Comfortably exceeds your chunk size | Silent truncation of your typical chunk |
| Dimensions | Smallest size that clears your accuracy bar | Chosen for leaderboard rank, not for storage reality |
| Multilingual coverage | Measured on your languages, including cross-lingual query to document | English-heavy training with token-level support elsewhere |
| Prefix or instruction | Documented query and passage prefixes, applied consistently | Prefixes ignored at ingest or query, which quietly degrades recall |
| Domain fit | Handles your jargon, codes, and document formats in your own tests | Only evaluated on public benchmark corpora |
| Provenance and version | Known publisher, pinned revision hash recorded with every vector | Untracked “latest” tag that can change under you |
| Fine-tuning path | Can be adapted on in-domain pairs inside your infrastructure | Adaptation only possible through a vendor service |
Score the shortlist across all rows rather than pass or fail on one, so the tradeoffs stay visible to whoever signs off.
How many embedding dimensions do you actually need?
Embedding dimensions map linearly to storage and memory, so the number is a budget decision as much as an accuracy decision. The table shows raw vector storage at float32 precision, before vector index structures, chunk text, and metadata are counted.
| Dimensions | Size per vector (float32) | 1M chunks | 10M chunks | 10M chunks at int8 |
|---|---|---|---|---|
| 384 | 1.5 KB | 1.5 GB | 15 GB | 3.8 GB |
| 768 | 3.1 KB | 3.1 GB | 31 GB | 7.7 GB |
| 1024 | 4.1 KB | 4.1 GB | 41 GB | 10 GB |
| 1536 | 6.1 KB | 6.1 GB | 61 GB | 15 GB |
| 4096 | 16.4 KB | 16.4 GB | 164 GB | 41 GB |
Two practical notes. An approximate-nearest-neighbour graph index adds overhead on top of the raw vectors, mostly per vector for the graph links rather than per dimension, but the stored vectors themselves still dominate the total, so a 4096-dimension model is not a marginal upgrade over a 1024-dimension one at ten million chunks. Quantization is the usual escape hatch: int8 cuts storage roughly fourfold and binary quantization far more, at a recall cost you should measure rather than assume. Several current model families are trained with Matryoshka representation learning, which lets you truncate a vector (1024 down to 512 or 256) with graceful rather than catastrophic degradation, a useful property when on-premise storage is fixed. For most enterprise corpora, 384 to 1024 dimensions is the sensible range.
Which open-weight embedding models run inside the perimeter?
Open-weight embedding families that regulated buyers commonly shortlist for on-premise RAG include the E5 family (multilingual-e5 in small, base, and large sizes, compact, and dependent on explicit query: and passage: prefixes), the BGE family from BAAI (including BGE-M3, with long input and dense, sparse, and multi-vector output from one pass), the GTE family (compact multilingual variants), Nomic Embed Text (long context and Matryoshka truncation), Qwen3-Embedding in 0.6B, 4B, and 8B sizes with multiple dimension options, and IBM’s Granite embedding models. All of these publish downloadable weights that can be served with no outbound call.
Licensing is the trap that catches teams late, because terms are not uniform across this space and can differ between versions and sizes of the same family, exactly as they do on the generation side, a pattern covered in how to choose an open-weight LLM for on-premise use. At least one widely cited multilingual embedding model is published under a non-commercial Creative Commons license. Verify the license text and the revision hash on the exact artifact you download, and record both in your change log.
How much should you trust MTEB and leaderboard scores?
MTEB and similar leaderboards are good for building a shortlist and poor for making the final call. The benchmark test sets are public, so leaderboard position partly measures how much a model has been tuned toward them. The corpora are general (web text, encyclopedia articles, academic abstracts, news) and look nothing like your circulars, discharge summaries, or loan policy manuals. The headline average blends classification, clustering, and summarization tasks that have no bearing on RAG retrieval, so read the retrieval subset for your languages instead. Gaps near the top of the table are often small enough to disappear on a specific corpus.
Use the leaderboard to pick three candidates. Then decide with a labelled golden set of your own questions and known-correct passages, scoring retrieval recall and precision per model, using the method in how to evaluate RAG accuracy. A day of labelling beats a month of benchmark reading.
How do you test multilingual and domain fit on your own corpus?
Test multilingual fit with the query pattern you will actually serve. Many enterprise deployments in India need cross-lingual retrieval: a question typed in Hindi against documents written in English. That is harder than same-language retrieval, and models that score well on English retrieval can degrade sharply on it. Build the test set from real bilingual questions and measure whether the correct English passage is retrieved for the Hindi query.
Test domain fit where embeddings are structurally weak: exact identifiers. Embedding models blur rare tokens, so two different policy numbers or clause references can land close together in vector space. If your corpus is dense with codes, account numbers, statute citations, or drug names, no embedding model fixes that alone, and the answer is hybrid retrieval, as set out in vector search versus keyword search for enterprise RAG. Two moves are usually cheaper than jumping to a larger embedding model: add a cross-encoder reranker, which can be swapped without touching the index, or fine-tune a modest embedding model on in-domain query and passage pairs inside your own infrastructure.
What does it cost to switch embedding models later?
Switching embedding models costs a full re-index, and planning for that is part of choosing well. The migration has a fixed shape:
- Estimate throughput first. Measure chunks per second for the new model on your actual GPU, then multiply by corpus size. This is a half-day experiment that prevents a multi-week surprise.
- Build in parallel. Vector spaces cannot be mixed, so the new index is built alongside the old one. Plan for roughly double the storage during migration.
- Re-tune retrieval. Top-k values, similarity thresholds, and hybrid fusion weights do not transfer between models. Assume they must be re-set.
- Re-validate before cutover. Run the golden test set against the new index and compare recall, precision, and citation correctness against the old numbers. Keep the old index until the new one wins.
- Handle air-gap logistics. In an air-gapped site the new weights must pass through media transfer and approval, often the longest step in the calendar.
The conclusion follows from the cost: choose an embedding model you can live with for a long time, and keep the volatile decisions (reranker, generation model, prompt) in the layers that are cheap to change.
Where does embedding model selection fit in a source-cited on-premise deployment?
Embedding model selection is one part of building a RAG system that stays inside your perimeter and can prove where every answer came from. Samvad AI, the secure source-cited RAG assistant from Teclops AI, runs inside your own infrastructure, answers only from your own documents, cites the exact source passage, and says plainly when an answer is not in your sources. It is permission-aware at the role and row level, multilingual, keeps a tamper-evident audit log, and deploys on-premise, air-gapped, or hybrid, switchable by configuration. If you want the embedding model chosen, evaluated on your own corpus, and served entirely within your network, our AI product development and consultancy services cover that work, or write to teclops.ai@gmail.com.
Frequently asked questions
What is the best embedding model for on-premise RAG?
There is no single best embedding model for on-premise RAG. The right pick is the smallest open-weight model that is licensed for commercial self-hosting, accepts your chunk length, covers your users' languages, and clears your accuracy bar on a golden test set built from your own documents.
Do you have to re-embed everything if you change the embedding model?
Yes. Vectors produced by two different embedding models are not comparable, so switching models means re-embedding every chunk and rebuilding the vector index. There is no incremental migration path: you build a parallel index, validate it against your test set, then cut over.
How many dimensions should an embedding model have for RAG?
For most enterprise RAG corpora, 384 to 1024 dimensions is the practical range. Storage scales linearly, so 1024-dimension float32 vectors take about 4 KB each, roughly 41 GB per 10 million chunks before index overhead, and larger vectors rarely repay that cost unless your own evaluation shows a real retrieval gain.
Can an embedding model run on CPU, or does it need a GPU?
Embedding models are far smaller than generation models, often in the 100 million to 600 million parameter range, so embedding a single short query runs acceptably on CPU at low concurrency. Bulk ingestion of a large corpus, and any later re-embedding, is much faster on GPU, so most on-premise deployments keep GPU capacity for indexing.
Are MTEB leaderboard scores reliable for choosing a RAG embedding model?
MTEB scores are useful for building a shortlist and unreliable as a final decision. The test sets are public, which invites overfitting, the corpora are general rather than your domain, and the headline average blends tasks unrelated to retrieval. Read the retrieval subset for your languages, then decide on your own labelled questions.