Get Started
Sizing Your Hardware
Before picking a model, figure out what actually fits in the memory you have.
The formula
Rough memory needed to load a model (ignoring KV cache and overhead):
memory (GB) ≈ parameters (billions) × bytes per parameterBytes per parameter by precision:
| Precision | Bytes / parameter |
|---|---|
| FP16 / BF16 | 2 |
| Q8 | ≈ 1 |
| Q5_K_M | ≈ 0.7 |
| Q4_K_M | ≈ 0.6 |
| Q3 | ≈ 0.4 |
Examples
- 7B model at FP16 → ~14 GB
- 7B model at Q4_K_M → ~4.2 GB
- 70B model at Q4_K_M → ~42 GB (needs a serious multi-GPU box, or heavy CPU offload)
Add ~10–20% on top for KV cache and runtime overhead at moderate context lengths — much more if running long contexts (32K+) or serving multiple concurrent requests. See Performance & Troubleshooting.
Practical starting points
| Your hardware | What to run |
|---|---|
| 8–16 GB VRAM (or Apple Silicon, 16 GB+ unified memory) | 7B–14B models at Q4_K_M |
| 24 GB VRAM | Up to ~30B at Q4_K_M, or 7–14B at higher precision with long context |
| CPU-only, 16–32 GB RAM | Small models (1–8B) at Q4 — expect a few tokens/sec, not tens |
Once you know your budget, head to Choosing a Tool to pick how you'll actually run it.