What server do you need to run an LLM in-house? GPU memory, KV cache and concurrent users

“Will this model run on our server?” is usually the first question from an organization that wants a private AI. The answer is almost always yes, as long as the GPU has enough memory. The more important question, and the one asked less often, is: “How many people can use it at the same time, and how long will each of them wait?”

This article is only about sizing the hardware. Most guides on running a model locally are about Ollama or LM Studio on a laptop. That’s fine for one person, but an organization with dozens of users needs a different calculation. The numbers below are based on the models’ published configurations and the cards’ spec sheets, and wherever a number is an estimate, I say so.

“Will it run?” is not the same as “How many users at once?”

When a language model runs on a GPU, the card’s memory (VRAM) has three main uses:

• Model weights: loaded once and shared by all users.
• KV cache (key-value cache): kept separately for every conversation being processed, and it grows with the length of the text.
• Overhead: intermediate activations, CUDA libraries and the like.

“Will it run?” is only about the first item. “How many users at once?” is about the second, and in practice that is the item that sets the hardware budget.

A “concurrent user” means a request being processed right now, not someone who has left the assistant window open. Little’s Law helps here: concurrent requests = arrival rate × duration of each response. If, at the busiest hour, 80 of 200 employees each ask a question every three minutes, about 0.44 requests arrive per second; with 20-second responses, that’s about 9 concurrent requests on average. Peaks are higher, so in the example in this article I use 20 as the baseline.

Weight memory: parameters times bytes

This is the easiest part of the calculation:

weights_bytes ≈ parameters × bytes_per_parameter

FP16 / BF16      → 2 bytes
INT8 / FP8       → 1 byte
4-bit (AWQ, GPTQ) → ~0.5 byte

In practice, 4-bit files are a little larger than the theoretical number, because some layers, such as the embedding layer and the output layer, stay at higher precision, and scaling factors are stored alongside the weights. For example, Qwen3-32B has about 32.8 billion parameters. Its theoretical 4-bit size is 16.4 GB, but the files of the official AWQ version on Hugging Face come to about 19.3 GB.

A 70-billion-parameter model at full precision needs two 80 GB cards just for its weights. A 4-bit 32B model fits on one 24 GB card, but leaves little room for users. Quantization usually costs a bit of quality. Measure that loss on your own data with an evaluation set, especially if you work in a language other than English such as Persian, rather than trusting English benchmark tables.

KV cache: the cost of each user

To generate each new token, the model needs the keys and values of all previous tokens in every layer. So they don’t have to be recomputed every time, they are kept in memory. The size of this memory per token comes from this formula:

kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_value

The 2 is for keys and values. Take the other numbers from the model’s config.json: num_hidden_layers, num_key_value_heads and head_dim.

Example: Qwen3-32B

According to its published configuration, this model has 64 layers and 64 attention heads for queries, but only 8 heads for keys and values, and each head has a dimension of 128. If the KV cache is kept in BF16, that is two bytes per value:

2 × 64 × 8 × 128 × 2 = 262,144 bytes ≈ 256 KiB per token

8,192 tokens × 256 KiB = 2 GiB  per conversation
20 conversations × 2 GiB = 40 GiB

In other words, 20 full conversations with an 8K-token context need twice as much memory as the model’s 4-bit weights. This is exactly what single-user guides don’t show you.

Why GQA matters so much

In Grouped-Query Attention (GQA), several query heads share one key-value head. In Qwen3-32B, every 8 query heads share one key-value head. If each of the 64 heads had its own keys and values, all the numbers above would be 8 times larger: 2 MiB per token, 16 GiB per 8K-token conversation, and 320 GiB for 20 users. Almost all recent open models use GQA, but check num_key_value_heads before you pick a model.

The interesting thing about this table is that the 70B model’s KV cache per token is only a little larger than the 32B model’s. The real gap between the two is in the weights.

You have two levers. First, an FP8 KV cache, which halves its size (in vLLM, the --kv-cache-dtype fp8 option); test its effect on quality. Second, a sensible cap on context length. Just remember that some languages use more tokens than English for the same text (in my experience, Persian uses about twice as many), so 8K tokens hold less text, and in RAG the retrieved documents take a large share of that budget.

Speed: prefill and decode are two separate problems

Answering each request has two phases. In prefill, the whole input (system prompt, question and retrieved documents) is processed at once, in parallel. This phase is limited by the card’s compute and determines the time to first token. In decode, tokens are produced one at a time, and for each token all the weights have to be read from memory. This phase is mostly bound by memory bandwidth.

The bandwidth rule

A rule of thumb for the ceiling on decode speed for a single user:

max_tokens_per_sec ≈ memory_bandwidth / bytes_read_per_step

Qwen3-32B-AWQ on L40S:     864 GB/s  / 19.3 GB ≈ 45 tokens/s
Qwen3-32B-AWQ on H100 SXM: 3350 GB/s / 19.3 GB ≈ 170 tokens/s

These are theoretical ceilings and real numbers are lower, but the logic matters: in decode, the card with more bandwidth is faster.

The KV cache is also read at every step, so it affects speed too. If 20 users have an average context of 4,000 tokens, each step reads about 20 GB of KV cache plus 19 GB of weights. On an L40S that means about 21 steps per second: about 21 tokens per second per user, and about 420 in total. This is also an optimistic estimate that ignores compute and overhead.

For prefill, the rule of thumb is about “2 × number of parameters” operations per input token. A 6,000-token RAG input on a 32B model is about 4×10¹⁴ operations, which takes anywhere from a fraction of a second to a few seconds depending on the card and server load. Prefix caching in vLLM avoids reprocessing a fixed system prompt for every request.

Continuous batching and PagedAttention

Because the decode bottleneck is reading the weights, advancing 20 requests in one step isn’t much more expensive than advancing one; the weights are read once and used for all of them. With continuous batching, finished requests leave and new ones join at every step, so nobody waits behind someone else’s long answer.

PagedAttention, the core idea behind vLLM, solves the memory problem: instead of reserving the maximum context length for every request, the KV cache is allocated in small blocks, like pages of an operating system’s virtual memory, and only as needed. The paper (Kwon et al., 2023) reports 2–4× the throughput of FasterTransformer and Orca at the same latency. The practical result: average usage matters, not each user’s worst case. And when memory fills up, vLLM doesn’t crash; requests wait in the queue or are temporarily set aside (preemption). Users see a slowdown, not an error.

Where Ollama and llama.cpp are enough

Ollama and llama.cpp are excellent for one person on a laptop: simple install, GGUF files, and running on CPU and GPU together. To be fair, the llama.cpp server has continuous batching and parallel slots, and Ollama accepts parallel requests through OLLAMA_NUM_PARALLEL (its docs say the default is 1, and memory grows with “parallel requests × context length”). But their focus is one machine and a few users. For dozens of concurrent users on a GPU, vLLM and SGLang, which are built for high throughput, usually deliver much higher total throughput. Don’t take my word for it; test with a load that looks like your real one.

SGLang is in the same class as vLLM, and its signature feature is RadixAttention: reusing the KV cache across requests that share a prefix. Hugging Face’s TGI, according to its GitHub page, is in maintenance mode and its repository was archived in March 2026; don’t start a new project on it.

A full calculation: an internal assistant for 200 employees

Assumptions

• 200 employees, sending at most 20 concurrent requests at the peak hour.
• The Qwen3-32B model with 4-bit AWQ quantization.
• A context cap of 8,192 tokens per conversation, including the system prompt, the RAG documents and the answer itself.
• vLLM as the inference engine.
• Numbers are rounded and the overhead is an estimate.

Memory calculation

weights (AWQ 4-bit)            ≈ 19 GB
KV cache, BF16: 20 × 2 GiB     ≈ 40 GB
overhead (estimate)            ≈ 4–6 GB
----------------------------------------
worst case, BF16 KV cache      ≈ 64 GB
worst case, FP8 KV cache       ≈ 44 GB   (KV: 20 × 1 GiB)

“Worst case” means all 20 users at once with a completely full context, which is rare; with PagedAttention, if conversations are half full on average, the KV cache is half as large. The choice is yours: buy for the worst case, or accept that during rare peaks some requests will wait briefly in the queue.

Which card?

For this scenario, I consider two 48 GB cards (L40S or RTX 6000 Ada) or one 80–96 GB card a sensible choice. A single 48 GB card is also enough to start with and for a pilot period, provided you use an FP8 KV cache and accept some queueing at peaks.

Gaming cards are cheaper and have good bandwidth, but they come with hidden costs. Their triple-slot coolers don’t fit most server chassis, direct card-to-card communication (P2P) isn’t officially supported, and since late 2017 the GeForce driver license has included a clause that does not permit data center deployment.

One card or two: tensor parallelism

In tensor parallelism, the matrices of each layer are split across the cards; each card holds half the weights and half the key-value heads. The memory and bandwidth of the two cards add up, so decode gets faster too. The cost is communication between the cards at every layer: small with NVLink (on the SXM versions of the A100 and H100), and larger over PCIe, but usually acceptable for two cards. The number of attention heads has to be divisible by the number of cards; Qwen3-32B, with 64 query heads and 8 key-value heads, works fine on 2, 4 or 8 cards.

vllm serve Qwen/Qwen3-32B-AWQ \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 32

The alternative is to run a complete, independent copy of the model on each card with a load balancer in front. No communication between the cards is needed, but each card has only its own memory. On a 48 GB card, after 19 GB of weights and the overhead, about 22 GB is left for the KV cache: about 11 full conversations in BF16 or 22 in FP8, per card.

When vLLM starts up, it prints the KV cache capacity in tokens to the log. Divide it by the context length and compare it with your own calculation.

So you know where these calculations come from: I’ve worked with these setups myself: one RTX 3090, two RTX 3090s, two RTX A6000s working together, and two RTX 5080s side by side. Each of them puts the same questions from this article in front of you: where the weights fit, how much is left for the KV cache, and how many concurrent users actually get answers.

The same scenario with a 4-bit Llama 3.3 70B: 40 GB of weights, 20 × 2.5 GB of KV cache plus overhead, meaning about 95 GB with BF16 and 70 with FP8. Two 48 GB cards with FP8 are just barely enough, and two 80 GB cards are comfortable.

Parts that usually get forgotten

• System RAM: my rule of thumb is at least as much as the total size of the model files, and preferably more, because the database, Elasticsearch and the containers need RAM too. For the example server above, 128 GB is a cautious choice.
• NVMe storage: loading 19 GB from an ordinary SATA SSD (about 0.5 GB/s) takes nearly 40 seconds; from NVMe, a few seconds. A few models and versions side by side quickly fill one or two terabytes.
• CPU and PCIe lanes: tokenization, request scheduling and the API run on the CPU. With several cards, make sure each one gets a full PCIe x16 link.
• Power and cooling: two L40S cards alone draw 700 W; with the rest of the server, that’s roughly 1.2–1.5 kW of continuous draw (an estimate). Data center cards such as the L40S and A100 have no fans and depend on the server chassis’s airflow.
• Embedding model and reranker for RAG: models such as bge-m3 or bge-reranker-v2-m3 have about 568 million parameters and take one or two gigabytes in FP16. They’re small, but on the same GPU they compete with the main model for memory and compute. Either reduce vLLM’s share with --gpu-memory-utilization or put them on a separate card. Putting them on the CPU is tempting but slow: in my own Persian RAG benchmark, the reranker took about 12 seconds per question on a CPU.

If you’re deploying in Iran

I work in Iran, where sanctions change the hardware side of a project. If you’re elsewhere, you can skip to the checklist; if your server will sit in Iran, or you’re working with a team there, these points matter.

Buying GPUs under sanctions. Data center cards are subject to export controls and reach Iran through the gray market and third countries. That means no official warranty, the card may be used or repaired, and delivery takes time. Before you pay, check the card with nvidia-smi and a stress test lasting several hours. Prices also change quickly; any number I wrote today would be stale by the time you read it. Get a quote close to the time of purchase.

Renting a GPU server inside or outside the country. If data must not leave the country, which is the case for most government and public organizations, a foreign server is out of the question; many foreign providers also refuse Iranian customers because of sanctions. Some Iranian data centers and cloud providers rent out GPU servers. Ask exactly which card it is, whether it’s dedicated or shared, who has root access, and what the contract commits to regarding your data.

Downloading weights and packages. Direct access to Hugging Face from inside Iran isn’t always stable, and you may need a workaround; plan for the same with Docker images and pip packages. The clean approach is to download the weights somewhere else and then transfer them, which you need to do anyway for a server without internet access. Compare the files’ SHA256 checksums with the values on Hugging Face, and prefer safetensors over pickle files, especially when a file comes from an unofficial mirror.

Power. With the power cuts of recent years, a UPS for a server drawing 1–1.5 kW isn’t an optional extra; it’s part of the design.

For renting GPU servers inside Iran, I use IranGPU, and my experience with their service has been better than with others. That’s only my personal experience, not an advertisement; before renting, compare the card, memory, data center location and support terms yourself.

Checklist before you buy

These calculations are a starting point, not a substitute for testing. If you can, run the same model for a few days on a rented server under a load similar to your real one before you buy, and measure per-user speed and time to first token. If you’d like help sizing or deploying a language model on your organization’s servers, see my page on AI consulting and implementation.