Self-Host an LLM: On-Premise Deployment Guide 2026
Run an LLM on your own hardware with zero data leaving your network. Covers GPU sizing, the serving stack, security controls, and offline updates.
Deploying an LLM on-premise means running the model, the retrieval pipeline, and the orchestration entirely on hardware you control, so no prompt, document, or answer ever leaves your network. Most self-hosting guides stop at the bare model: an inference server on a GPU that answers from its training data. A real enterprise deployment is the full stack inside your perimeter: an inference server that hosts open-weight model files, a retrieval layer (an embedding model plus a vector database) that grounds answers in your own documents with source citations, an orchestration layer that builds prompts and enforces rules, and a governance layer for permission-aware access control and audit logging. That full source-cited pattern is what Teclops AI packages as Samvad AI, and it is the pattern this guide walks through: GPU sizing first, then the deployment process in seven explicit steps.
GPU sizing quick reference
VRAM, not raw compute, is the binding constraint for a self-hosted LLM. The quick-reference table below maps model classes to their approximate VRAM footprint (weights plus modest cache headroom) and a workable GPU configuration. Figures follow the standard bytes-per-parameter math (about 0.5 GB per billion parameters at 4-bit, 2 GB at 16-bit) detailed in our air-gapped LLM hardware and VRAM sizing guide.
| Model class | VRAM at 4-bit (quantized) | VRAM at 16-bit (unquantized) | Example GPU config (4-bit) | Rough concurrent users |
|---|---|---|---|---|
| 3-9B | ~6 GB | ~18 GB | one 24 GB GPU | tens |
| 24-32B | ~20 GB | ~68 GB | one 24 GB GPU (tight) or 48 GB | ~10-20 |
| 70B dense | ~44 GB | ~150 GB | one 48-80 GB GPU | ~5-15 |
| ~109B MoE (17B active) | ~66 GB | ~230 GB | one 80 GB GPU (tight) or two | single digits |
The concurrent-user column is a planning estimate only: each simultaneous request consumes VRAM for its key-value cache, and that cache grows with context length, so long retrieved passages cut concurrency sharply. Reserve 20 to 40 percent of VRAM beyond the weights for cache and activations, and validate under your real load. Before committing to hardware at all, sanity-check the economics for your query volume against the on-prem LLM TCO breakdown of GPU capex versus cloud API cost, because steady enterprise load amortizes GPUs far better than low, spiky usage does.
What is on-premise LLM deployment?
On-premise LLM deployment is the practice of hosting a large language model and its retrieval pipeline on infrastructure your organization owns and controls, rather than calling a model API run by an outside provider. The model weights, the document index, and every prompt and response stay inside your perimeter. This is the baseline requirement for regulated institutions that cannot let sensitive text cross an organizational boundary.
Why deploy an LLM on-premise instead of using a cloud API?
On-premise LLM deployment keeps your prompts, documents, and answers inside your own perimeter, which matters when data sovereignty, regulatory scope, or contractual confidentiality forbid sending text to an external provider. Banks, hospitals, universities, and government bodies often cannot let regulated data cross an organizational boundary at all. The tradeoff is that you own the hardware, the model lifecycle, and the operational burden that a managed API would otherwise absorb. For a fuller comparison of the economics and control involved, see on-prem vs cloud LLM.
How do you deploy an LLM on-premise, step by step?
An on-premise LLM deployment follows seven steps, from the isolation decision through to the offline update process:
- Choose the deployment topology (air-gapped, connected, or hybrid).
- Size the GPU hardware for your models and concurrency.
- Select an open-weight model.
- Stand up the inference server and serving engine.
- Build the private RAG retrieval layer.
- Wrap the pipeline in access control and audit logging.
- Establish the offline model and software update process.
Each step is detailed below.
Step 1: Choose the deployment topology
The topology of an on-premise LLM deployment decides how isolated it is from any outside network, and it shapes every later step. Most regulated teams pick one of three patterns and switch by configuration rather than rebuilding:
- Air-gapped: no network path to the internet at all. Model weights, embeddings, vector store, and orchestration run fully offline. Updates arrive as signed files through controlled transfer. Highest assurance, highest operational discipline.
- Connected (on-prem, network-attached): runs on your own hardware but can reach internal services and, where policy allows, fetch updates online. Simpler operations, broader exposure.
- Hybrid: sensitive workloads stay isolated while less-sensitive components (for example, model updates or non-confidential integrations) use a controlled connection.
A well-designed system lets you choose air-gapped, connected, or hybrid by configuration, so the same architecture serves a maximum-security unit and a less-restricted department without separate builds.
Step 2: Size the GPU hardware
A self-hosted LLM is sized primarily by GPU memory (VRAM), not raw compute. The model weights must fit in VRAM, and each concurrent request also consumes VRAM for its key-value cache, so total memory drives the decision more than clock speed. Start from the quick-reference table above, then adjust for four factors:
- Model size: larger parameter counts need more VRAM. Quantizing weights (lower numeric precision) reduces the footprint at some quality cost.
- Concurrency: more simultaneous users means more key-value cache, so plan VRAM for peak parallel requests, not just the model itself.
- Context length: longer prompts and retrieved passages enlarge the cache per request.
- Retrieval components are light: the embedding model and vector database are far less demanding than the LLM and often run on CPU or share existing hosts.
Right-size for your real concurrency rather than a theoretical maximum, and leave headroom for the cache.
Step 3: Select an open-weight model
Model selection on-prem favors open-weight models you can download once, store as files, and load locally. Because the weights are static files, an air-gapped LLM stays fully functional offline indefinitely: “no internet” constrains how you update, not whether the system runs. Weigh license terms, task fit on your own documents, and the VRAM budget from Step 2, and keep at least one fallback model validated so a swap is a file operation rather than a procurement event.
Step 4: Stand up the inference server and serving engine
The inference server is the process that loads the model weights onto the GPUs and turns prompts into text, and the serving engine you pick determines how far your hardware stretches. The decision comes down to concurrency and operational surface: a batching engine built for many simultaneous users typically extracts far more throughput from the same GPU, while lighter runtimes are simpler to operate for pilots, single-user work, and CPU-only tests. For a concrete recommendation by team size and hardware, see our comparison of vLLM vs Ollama vs llama.cpp as an on-prem inference server.
Step 5: Build the private RAG retrieval layer
A private RAG (retrieval-augmented generation) architecture is a system that grounds an LLM’s answers in your own documents while running every part inside your perimeter. It has five layers, and the pattern is the same whether you build it yourself or adopt a packaged system.
| Component | Role | Typical footprint |
|---|---|---|
| Inference server | Hosts the LLM, generates text from prompts | GPU, the heaviest component |
| Embedding model | Converts documents and queries into vectors | GPU or CPU |
| Vector database | Stores embeddings, returns nearest matches | CPU, disk |
| Retrieval + orchestration | Finds relevant passages, assembles the prompt, calls the model | CPU |
| Access control + audit | Filters by permission, logs every query and answer | CPU |
The orchestration layer is where retrieval, prompting, and policy meet: it embeds the user’s question, queries the vector database for relevant passages, builds a grounded prompt, and sends it to the inference server. Keeping retrieval on-prem is what makes the pipeline “private RAG”: your documents are indexed and queried locally, and no passage is shipped to an outside service.
Step 6: Wrap the pipeline in access control and audit logging
Access control and audit logging are not optional add-ons in a regulated on-prem deployment; they decide what each user may retrieve and create a defensible record of every answer. Two mechanisms matter most:
- Permission-aware retrieval: enforce role-level and row-level rules at retrieval time so a user can never be shown a passage they are not cleared to see. The filter must happen before the model sees the text, not after.
- Tamper-evident audit log: record every query, the sources retrieved, and the answer returned, in a log that detects alteration. This is what lets you reconstruct, months later, exactly what the system told someone and why.
Grounding answers in retrieved sources also lets the system cite the exact passage behind each response and say plainly when an answer is not in your documents, which is essential for review in regulated settings. The Teclops AI security architecture page covers these controls in more depth.
Step 7: Establish the offline update process
The update process replaces or supplements the model and software files under change control, even with no internet access:
- Validate the new weights or runtime in a staging environment against your own evaluation set.
- Sign and transfer the files into the isolated zone using approved media or a one-way transfer mechanism (such as a data diode).
- Load as a new version alongside the current one, so you can compare and roll back.
- Promote only after the new model passes your checks.
For the full day-two procedure, including the staging enclave, signed artifacts, and CVE patching cadence, see how to patch an air-gapped AI system offline.
How does Samvad AI package on-premise LLM and RAG deployment?
Samvad AI is Teclops AI’s secure, source-cited RAG assistant that answers only from your own documents and packages the components above so teams do not assemble them by hand. Samvad AI deploys on-premise, air-gapped, or hybrid (switchable by configuration), is permission-aware at role and row level, cites the exact source passage for every answer, states plainly when something is not in your sources, supports English, Hindi, and more, and keeps a tamper-evident audit log. If you would rather have the inference, retrieval, and governance layers built into your own infrastructure as a service engagement, Teclops AI’s AI product and automation services cover that, all deployable inside your perimeter. Reach the team at teclops.ai@gmail.com.
Frequently asked questions
How do you deploy an LLM on-premise?
To deploy an LLM on-premise, you run an inference server loaded with open-weight model files on your own GPU hardware, place a vector database and embedding service alongside it for retrieval, and add an orchestration layer that handles prompts, access control, and logging. Everything stays inside your network perimeter, with no calls to an external model API.
Can you run a RAG system air-gapped with no internet?
Yes. An air-gapped RAG system runs entirely offline because the model weights, embedding model, vector database, and orchestration all live on local hardware. Model and software updates are delivered as signed files through a controlled transfer process (physical media or a one-way data diode) rather than downloaded live from the internet.
What hardware do you need to self-host an LLM for an enterprise?
Self-hosting an LLM for an enterprise requires GPUs with enough total VRAM to hold the model weights plus the key-value cache for concurrent requests. Larger models and higher concurrency need more VRAM, which usually means multiple GPUs. Retrieval components (vector database, embeddings) are lighter and often run on CPU or share the same hosts.
What components make up a private RAG architecture?
A private RAG architecture has five parts that all run inside your perimeter: an inference server hosting the LLM, an embedding model that turns text into vectors, a vector database storing those vectors, a retrieval and orchestration layer that finds relevant passages and builds the prompt, and access control plus audit logging wrapped around the whole pipeline.
How do you update an on-premise LLM without internet access?
You update an on-premise LLM by validating new model weights or software in a staging environment, then transferring the signed files into the air-gapped zone through approved media or a one-way transfer mechanism. The new model is loaded as a separate version so you can test, compare, and roll back without disrupting the running system.