llama.cpp

Use this when you want full control — custom quantization, unusual hardware, or the extra 10–20% performance over Ollama's wrapper.

Build from source

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# CPU-only build
cmake -B build
cmake --build build --config Release -j 8

# For NVIDIA GPU (CUDA) acceleration instead:
# cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=OFF
# cmake --build build --config Release -j 8

Binaries land in build/bin/ — the two you'll use most are llama-cli (interactive terminal chat) and llama-server (HTTP server). See the official build docs.

Get a model

Download a pre-quantized GGUF from Hugging Face (search "<model name> GGUF"), or convert a safetensors model yourself:

# Convert an original HF model to GGUF (FP16), then quantize it
python convert_hf_to_gguf.py /path/to/hf-model --outfile model-f16.gguf
./build/bin/llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Run it

# Interactive terminal chat
./build/bin/llama-cli -m model-q4_k_m.gguf -p "You are a helpful assistant." -cnv

# HTTP server with OpenAI-compatible API on port 8080
./build/bin/llama-server -m model-q4_k_m.gguf -c 8192 --port 8080

-c 8192 sets the context window in tokens; -ngl N (not shown above) offloads N layers to GPU if you built with CUDA/Metal support — set it high (e.g. 99) to push as much as possible onto the GPU.