Reference

Performance & Troubleshooting

Tuning knobs for when things are slow, and fixes for the errors you'll hit most often.

Performance tuning

GPU layer offloading (-ngl in llama.cpp) — if the whole model doesn't fit in VRAM, offload only some transformer layers to GPU and leave the rest on CPU. More layers on GPU = faster, until you run out of VRAM.
Context length vs. memory — every doubling of context roughly doubles KV cache size at a given batch size. Don't request a bigger context window than you need.
Quantization level — drop from Q5/Q6 to Q4_K_M if memory constrained; the quality loss is usually smaller than expected, especially for straightforward tasks.
Batch size / concurrent requests — irrelevant for single-user chat, critical for a server — this is the whole reason vLLM exists over naive serving.
Speculative decoding — an advanced technique (supported by llama.cpp and vLLM) where a small "draft" model proposes several tokens and the large model verifies them in one pass, often 1.5–2x faster with no quality loss.

Troubleshooting

"CUDA out of memory"

Drop to a smaller quantization, reduce --max-model-len/-c context size, or offload fewer layers to GPU.

Extremely slow generation on CPU

Expected — CPU inference is 10–50x slower than GPU for the same model. Use a smaller model or get GPU offload working.

Model output looks garbled or repetitive

Check you're using the correct prompt template for that model — chat models expect a specific format wrapping user/assistant turns (tokenizer.apply_chat_template in transformers, or automatic in Ollama/llama-server when using the right GGUF).

Ollama model won't load / "model not found"

Run ollama list to confirm the exact tag; tags are case-sensitive and versioned.

vLLM won't start

Almost always a CUDA/driver version mismatch or insufficient --gpu-memory-utilization headroom; check nvidia-smi first.