On-Prem LLM High Availability: N+1 or Weeks Down

On-premise LLM high availability means running the same model on two or more stateless inference replicas behind a health-checking load balancer, with spare GPU capacity racked in advance because replacement hardware arrives in weeks. Everything genuinely stateful (weight files, source documents, the vector index, the audit log) lives on storage you back up and have practiced restoring. Cloud failover advice does not transfer to an air-gapped network: there is no second region and no fallback API key. This guide covers the on-prem failure ladder, the stateless serving pattern, whether a standby GPU node pays for itself at each service level, backup and restore, degraded mode, and how to test failover.

What actually fails in an on-premise LLM deployment?

On-premise LLM failures form a ladder, cheap and frequent at the top, rare and expensive at the bottom. Each rung needs a different mechanism, and a design that only handles process crashes will still take a multi-week outage when a GPU dies.

Failure	What you observe	Blast radius	What saves you
Serving process crash or CUDA out-of-memory	Connection refused, restart loop, 500s on long-context requests	1 replica, seconds to minutes	Process supervisor restart, conservative max-context and memory-utilization settings
Model load failure after restart	Replica never reports ready, checksum or config mismatch	1 replica, until fixed by hand	Pinned artifact versions, recorded checksums, staged promotion of new weights
GPU fault (uncorrectable ECC, Xid error, card off the bus, thermal shutdown)	Requests hang, driver errors in kernel logs, card missing from device enumeration	1 node, until the card is replaced	Spare capacity: a GPU reset or reboot may clear it and may not
Node loss (PSU, mainboard, kernel panic, rack power or cooling)	Node unreachable, all health checks fail	1 node, hours to weeks	N+1 replicas plus a load balancer that drains the dead node
Storage loss (weights volume or vector database volume)	Model will not load, or retrieval returns zero passages	Whole service	Backed-up source documents plus an offline artifact repository inside the perimeter
Load balancer or network loss	Every replica healthy, nothing reachable by clients	Whole service	2 proxy instances sharing a floating virtual IP

The distinction that sets your budget: the top three rungs are software problems with software answers, and the bottom three are procurement problems.

Why is cloud failover advice useless in an air-gapped network?

Standard LLM resilience guidance assumes a second provider: a fallback API key, another region, a retry against a different endpoint. In an air-gapped or on-premise deployment none of those exist. The failure domain is a room, sometimes a single rack, and every layer of redundancy is hardware bought in advance.

You cannot buy capacity during an incident. GPU replacement, whether RMA or fresh procurement, runs on lead times measured in weeks, and for current data center accelerators often months. Redundancy has to be racked before the failure.
There is no elastic scale-out. Peak concurrency is fixed by the cards you own, so a lost node is a capacity event as well as an availability event, and the survivors queue.
Recovery artifacts must already be inside the perimeter. You cannot re-download a 140 GB weight set from a public model hub across an air gap. An internal artifact repository holding the exact revisions you run is a prerequisite.

The mental model to carry: an on-premise high availability budget is set by hardware lead time, not by mean time between failures. The question is never whether a GPU will fail, but how many working days the service can be down while procurement runs.

How do you make LLM inference highly available?

Make the inference tier stateless and put a health-checking load balancer in front of it. This is the highest-value architectural decision in on-premise LLM operations, and it works because an LLM serving process is almost pure compute.

Stateless, safe to lose and recreate:

The serving process itself, whether vLLM, another engine, or a container of one.
The attention key-value cache, which is ephemeral per-request GPU memory. Never replicate it. If a replica dies mid-generation, fail that request and let the client retry.
Model weights loaded into VRAM, which are only a copy of a file on disk.

Stateful, must be backed up and restorable:

Weight files plus tokenizer and config artifacts.
The vector index and its metadata store.
Source documents, the ground truth behind the index.
Conversation history, permission and role mappings, and the audit log.

Deploy N+1 identical replicas, each holding the full model. Two servers running one model beats one server running two models, because the second arrangement is a capacity plan, not a redundancy plan. If a model needs tensor parallelism across several cards in one chassis, treat that chassis as one failure unit and add a second. Size each node first using the air-gapped LLM hardware and GPU VRAM sizing guide, because a replica without KV-cache headroom fails under exactly the load that follows a node loss.

How do you load balance multiple vLLM servers?

Put HAProxy, an equivalent proxy, or a Kubernetes Service in front of identical vLLM replicas that each expose the same OpenAI-compatible API and a /health endpoint. Since every replica holds the same weights and keeps no cross-request state, any request can go to any replica and no session affinity is needed.

Use least-connections, not round-robin. Request durations vary by orders of magnitude between a one-line answer and a long summarization, so round-robin piles long generations onto one replica while another idles.
Health-check the model, not the port. A TCP check only proves a socket is open, so it can still pass while a GPU has faulted and requests hang. Poll /health and, periodically, a cheap real completion.
Set streaming-aware timeouts. Token-by-token server-sent events look like a stalled response to default proxy settings. Use long client and server timeouts with a first-token timeout as the real liveness signal.
Drain, do not sever. For maintenance and model upgrades, stop new requests and let in-flight generations finish: a readiness probe going false plus a termination grace period longer than your maximum generation.
Make the load balancer redundant. Two proxy instances with a floating virtual IP, or you have moved the single point of failure up one layer.

The engine matters less than the interface. Any server speaking an OpenAI-compatible API fits this pattern, which is one reason to pick on operational fit using the vLLM vs Ollama vs llama.cpp inference server comparison.

Is a standby GPU node worth the cost?

A standby GPU node is worth buying when the cost of the outage it prevents exceeds the amortized cost of hardware that mostly idles. Tier the decision by the service level you are willing to promise, and write that promise down before purchasing.

Service level	Promise to users	Minimum architecture	Spare hardware	GPU nodes bought
Best effort (internal pilot)	“Usually available, no guarantee”	1 node, supervised restarts, documented rebuild	None: accept a multi-week outage during RMA	1 node, 0 idle
Business hours (departmental)	“Available 9 to 6 on working days”	2 replicas behind 1 load balancer	Cold spare card or node on the shelf	2 nodes, 1 recoverable as burst capacity
24x7 (customer-facing or clinical support)	“Always on, published recovery time”	3 or more replicas across separate racks and power feeds, 2 proxies	N+1 live plus a cold spare, second site if the site is the failure domain	3 or more nodes, at least 1 idle at all times

Two ways to make the spare earn its keep. First, run the redundant node as preemptible burst capacity for batch work: overnight document embedding, evaluation runs, fine-tuning jobs. Second, be honest that N+1 must survive peak: if two nodes each run at 80 percent of capacity, losing one does not degrade the service, it collapses it. Feed these node counts into the on-prem LLM TCO analysis, since redundancy is the line item most often missing from a self-hosting business case.

How do you back up and restore model weights and the vector index?

Model weights are large static artifacts; the vector index is a rebuildable derivative of your source documents. Confusing the two produces recovery plans that quote the wrong number.

Model weights. Immutable and identical across replicas. Keep every version you run in an internal artifact repository with recorded checksums. Restore is a file copy, so recovery runs at local disk and network speed. Never plan to re-fetch from a public source: across an air gap you cannot, and on a connected network the exact revision may have moved.
The vector index. Back it up for speed, but treat it as reconstructible. The irreplaceable inputs are the source documents, the ingestion and chunking configuration, and the pinned embedding model version. Lose that version and a rebuilt index will not match your evaluation results, because embeddings from different models are not comparable.
Governance data. Audit logs, permission mappings and conversation history are small, genuinely irreplaceable, and belong on the standard database backup schedule with restores tested like any other database.

Publish the vector index rebuild time as your recovery time objective, not the model reload time. Re-embedding consumes the same GPUs that serve traffic, so measure it once on the real corpus and quote the measured number.

What does degraded mode look like for an on-prem LLM?

Degraded mode is a deliberately reduced LLM service that stays truthful and useful when full capacity is gone, and designing it costs far less than buying enough hardware never to need it. The system should get slower or simpler under failure, never wrong or blank.

Queue and admit. Hold requests in a bounded queue with an honest wait estimate rather than timing out. Users tolerate slow far better than broken.
Shed expensive paths first. Cap maximum output length, cut retrieved-passage counts, and stop batch and background jobs before degrading interactive answers.
Fall back to a smaller model. A quantized small model on a CPU server or older workstation GPU answers routine questions. Label those answers in the interface so nobody mistakes fallback output for full-quality output.
Retrieval-only mode. If no generation capacity survives, keep search running. Returning exact source passages without a generated summary is still a working product, and for a source-cited assistant it is often most of the value.

The rule that keeps degraded mode safe: never silently substitute a weaker model or a truncated retrieval set. People making regulated decisions need to know which mode answered them, and the audit log needs to record it.

How do you test on-prem LLM failover before you need it?

Test failover on a scheduled day, in production or a production-identical environment, with the team watching, and record the measured time for every step. An untested plan is a document, not a capability.

Kill the serving process on one replica during working hours. Confirm the load balancer marks it down, in-flight requests fail cleanly, and clients retry successfully.
Power off a whole node. Confirm survivors carry peak load, not average load, and watch queue depth and time to first token.
Force a health check to fail during a long streaming generation. Confirm draining works and the response completes rather than being cut mid-sentence.
Fail the load balancer. Confirm the virtual IP moves and clients reconnect.
Restore weights from the offline artifact repository onto a blank node and time the full path to a serving replica.
Rebuild the vector index from source documents in staging and time it. That number is your true recovery time objective.
Run degraded mode for a full working day and collect feedback on whether the fallback labeling was clear.

Publish the observed numbers alongside your service level. A recovery time you have measured is defensible to an auditor; an estimate in a policy document is not.

Who builds the redundant serving tier if you do not want to assemble it?

High availability for on-premise inference is an architecture problem (stateless replicas, health-checked routing, spare capacity sized to a written service level, tested restore) rather than a product in a box. Samvad AI, Teclops AI’s secure, source-cited RAG assistant, is built for this deployment shape: it runs on-premise, air-gapped or hybrid by configuration, answers only from your own documents with the exact source passage cited, states plainly when an answer is not in your sources, and keeps a tamper-evident audit log, so retrieval and governance stay intact when a node is lost. See the Samvad AI on-premise RAG assistant for how the retrieval layer is deployed.

If you want the serving tier, redundancy plan and restore procedure designed and built inside your own infrastructure, Teclops AI’s AI product development and automation services cover that work. Reach the team at teclops.ai@gmail.com.

Frequently asked questions

How do you achieve high availability for a self-hosted LLM?

Run identical copies of the model on two or more stateless inference replicas behind a load balancer that health-checks each replica and pulls failures out of rotation automatically. Keep the stateful parts, meaning weight files, source documents, the vector index and the audit log, on separately backed-up storage, and rack spare GPU capacity before the failure rather than after it.

Can you load balance multiple vLLM servers?

Yes. Each vLLM server exposes the same OpenAI-compatible HTTP API and a /health endpoint, so HAProxy, an equivalent proxy or a Kubernetes Service can spread requests across identical replicas with no session affinity. Choose least-connections over round-robin, since one-line answers and long summarizations differ in duration by orders of magnitude, and raise proxy timeouts so streamed tokens are not cut off.

Do you need a standby GPU node for an on-premise LLM?

You need spare GPU capacity proportional to the service level you have promised in writing, which is not always a full standby node. A best-effort internal pilot can absorb a multi-week wait for a replacement card. Any service promising business-hours or 24x7 availability needs N+1 live capacity, because GPU procurement and RMA lead times run to weeks and, for in-demand data center accelerators, often far longer.

What is the real recovery time after an on-prem LLM node failure?

Restoring a stateless inference replica takes minutes, since it is a file copy of the weights onto healthy hardware plus a model load. The slow path is retrieval: rebuilding a lost vector index means re-embedding the whole source corpus on the same GPUs you need for serving, which on a large corpus runs far longer than a model reload, so that measured rebuild time is the number a disaster recovery plan should publish.

Does the KV cache need to be replicated between LLM servers?

No. The attention key-value cache is ephemeral per-request GPU memory, and replicating it across servers adds cost and complexity for no resilience benefit. When a replica dies mid-generation, the correct behavior is to fail that single request and let the client retry elsewhere, which is precisely why an LLM serving tier is easy to make redundant.