GPU calculator
How much GPU does your model need?
Pick a model, precision, context length and concurrency. AWSBee computes memory from each model's architecture and ranks every NVIDIA instance type by fit and cost.
GPU memory needed
203.2 GBof 288.0 GB usable on p4d.24xlarge
- Weights141.2 GB
- KV cache42.9 GB
- Runtime overhead19.1 GB
- Usable memory (90% of VRAM)
Cheapest instance that fits
p4d.24xlarge · 8× NVIDIA A100 40GB
~41 tok/s per stream · up to 23 full-context requests · ≈ $21.96/h
All NVIDIA GPU instances
Approximate on-demand list prices for us-east-1. Prices differ by region and change over time; check the official EC2 pricing page before launching. Throughput is a memory-bandwidth estimate, not a benchmark.
| Instance | GPUs | Usable | Max conc. | Tok/s (1 stream) | $/hour | $/1M out tok* |
|---|---|---|---|---|---|---|
p4d.24xlarge | 8× A100 40GB | 288 GB | 23 | 41 | 21.96 | 23.70 |
g6e.48xlarge | 8× L40S | 346 GB | 34 | 23 | 30.13 | 58.52 |
p5.48xlarge | 8× H100 80GB | 576 GB | 77 | 89 | 55.04 | 27.57 |
g4dn.xlarge Needs 192.7 GB, 14.4 GB usable | 1× T4 | 14 GB | — | — | 0.53 | — |
g6.xlarge Needs 192.7 GB, 21.6 GB usable | 1× L4 | 22 GB | — | — | 0.81 | — |
g6.2xlarge Needs 192.7 GB, 21.6 GB usable | 1× L4 | 22 GB | — | — | 0.98 | — |
g5.xlarge Needs 192.7 GB, 21.6 GB usable | 1× A10G | 22 GB | — | — | 1.01 | — |
g5.2xlarge Needs 192.7 GB, 21.6 GB usable | 1× A10G | 22 GB | — | — | 1.21 | — |
g6e.xlarge Needs 192.7 GB, 43.2 GB usable | 1× L40S | 43 GB | — | — | 1.86 | — |
g6e.2xlarge Needs 192.7 GB, 43.2 GB usable | 1× L40S | 43 GB | — | — | 2.24 | — |
g4dn.12xlarge Needs 197.2 GB, 57.6 GB usable | 4× T4 | 58 GB | — | — | 3.91 | — |
g6.12xlarge Needs 197.2 GB, 86.4 GB usable | 4× L4 | 86 GB | — | — | 4.60 | — |
g5.12xlarge Needs 197.2 GB, 86.4 GB usable | 4× A10G | 86 GB | — | — | 5.67 | — |
g6e.12xlarge Needs 197.2 GB, 172.8 GB usable | 4× L40S | 173 GB | — | — | 10.49 | — |
g6.48xlarge Needs 203.2 GB, 172.8 GB usable | 8× L4 | 173 GB | — | — | 13.35 | — |
g5.48xlarge Needs 203.2 GB, 172.8 GB usable | 8× A10G | 173 GB | — | — | 16.29 | — |
* At 8 concurrent streams with continuous batching, fully utilized. Tensor parallelism uses up to 8 GPUs for this model.
How the numbers are computed
- Weights = parameters × bytes per parameter (2 for BF16/FP16, 1 for FP8).
- KV cache = 2 × layers × KV heads × head dimension × 2 bytes, per token, times context × concurrent requests.
- Overhead = 1.5 GB per GPU for the CUDA context and graphs, plus 5% of weights for activations.
- Usable memory is 90% of VRAM, matching vLLM's default
--gpu-memory-utilization. - Throughput estimates assume decoding is memory-bandwidth bound: each token reads the active weights once, at ~55% of peak bandwidth.