AWSBee
← All articles

November 16, 2026 · 6 min read

How much GPU memory does an LLM actually need?

Model weights are only part of the bill. A simple way to estimate weights, KV cache and overhead before you pick a GPU.

The first question every team asks when moving off a hosted AI API is which GPU to rent. The usual answer, "parameters times two bytes", gets you the size of the weights, but a model that only just fits its weights will fall over on the first long prompt.

Three things share the GPU

  • Weights: parameters × bytes per parameter. An 8B model in 16-bit precision needs about 16 GB; the same model quantized to 8-bit needs about 8 GB.
  • KV cache: the memory that holds attention keys and values for every token in every active request. It grows with context length and with the number of requests served at once.
  • Overhead: the CUDA context, activations and the inference server's own buffers. Budget 10–15% of the card.

Estimating the KV cache

Per token, the cache needs 2 × layers × KV heads × head dimension × bytes per value. For a Llama 3.1 8B style model (32 layers, 8 KV heads, head size 128, 16-bit) that is about 128 KB per token. Eight concurrent requests at 8,000 tokens each is 64,000 tokens, or roughly 8 GB of cache on top of the weights.

That is why an 8B model with long contexts is comfortable on a 24 GB card such as the NVIDIA L4 or A10G, but tight on a 16 GB T4. A 70B model needs about 140 GB for 16-bit weights alone, so it is split across several GPUs with tensor parallelism, for example 4× L40S or 2× H100 with 8-bit weights.

A practical rule

Pick the smallest card where weights take no more than about 60% of memory. The remaining 40% becomes KV cache, which is what decides how many users you can serve at once. Inference servers like vLLM use whatever is left automatically, so more headroom directly means more throughput.

Our sizing engine does this math for every model and GPU pair, including tensor parallelism, and recommends the cheapest instance that fits your context length.

Want to see this with your own model?

Book a demo