Concepts & Glossary

An LLM is, mechanically, a huge set of numbers (parameters, also called weights) arranged in a neural network architecture — almost always a Transformer today. "Running a model" means loading these numbers into memory and doing matrix multiplication with your input. Model size is described by parameter count: "7B" means 7 billion parameters.

Models read tokens — sub-word chunks produced by a tokenizer — not raw characters or words. "Programming" might become Program + ming. Roughly 1 token ≈ 0.75 English words. Every input and output is measured in tokens; this is the unit that drives both memory use and cost.

The maximum number of tokens (input + output combined) a model can "see" at once. A 32K context window model forgets anything beyond the last ~32,000 tokens. Larger windows cost more memory — see KV cache, below.

Training builds the weights (extremely expensive, many GPUs, done once by a lab). Inference uses the weights to generate output (much cheaper — this is what you do locally). This entire guide is about inference; you never train a model from scratch here.

A lighter training step that adjusts an already-trained model on a smaller, task-specific dataset. Distinct from both training and inference — worth knowing the term, not covered in depth in this guide.

The single most important concept here. Weights are normally stored as 16-bit floats (FP16/BF16). Quantization compresses each weight to fewer bits — 8, 5, 4, or fewer — trading a little quality for a lot less memory and often faster inference. A 7B model at FP16 needs ~14 GB; quantized to 4-bit it needs roughly ~4 GB. This is what makes running a serious model on a consumer GPU possible at all.

Names like Q4_K_M or Q8_0: the digit is bits per weight. K means "K-quants" — grouped quantization with per-group scaling that minimizes rounding error, instead of one flat precision across the model. The suffix (_S/_M/_L) indicates how aggressively different layers are quantized; _M keeps extra precision on the layers that matter most and is almost always the right default.

Practical rule of thumb: Q4_K_M is the standard "good enough for almost everything" choice (~75% smaller than FP16, ~3–4% quality loss). Q5_K_M/Q6_K if you have memory to spare. Q8_0 is close to lossless but barely smaller than FP16.

VRAM is your GPU's dedicated memory; system RAM is your CPU's. GPU inference is much faster than CPU inference, but limited by VRAM size (consumer GPUs: 8–24 GB+; Apple Silicon shares unified memory between CPU and GPU, which is why Macs punch above their weight here). If a model doesn't fit in VRAM: quantize further, use a smaller model, or split layers between GPU and CPU ("offloading" — slower, but works).

During generation, the model caches intermediate attention calculations ("key" and "value" vectors) for every token already processed, so it doesn't redo that work per new token. This cache grows with context length and is a second major memory consumer beyond the weights — why long conversations use much more memory than the weight file size alone suggests.

Controls over how the model picks the next token from its probability distribution:

• Temperature — randomness. 0 = deterministic/greedy; higher (0.7–1.0) = more varied output.
• Top-p (nucleus sampling) — sample only from the smallest set of tokens whose cumulative probability exceeds p (e.g. 0.9).
• Top-k — sample only from the k most likely next tokens.
• Repetition/frequency penalty — discourages the model from repeating itself.

Latency is time-to-first-token and time-per-token for one request (what matters for a chat UI). Throughput is total tokens/sec across many concurrent requests (what matters for serving many users). Tools optimized for one are often bad at the other — the axis that distinguishes the tools covered in Choosing a Tool.

• safetensors — the standard Hugging Face distribution format: a safe, fast-loading container for raw FP16/BF16 weights.
• GGUF — a format built for llama.cpp, purpose-built for quantized models and CPU/consumer-GPU inference. Single self-contained file including tokenizer and metadata.
• ONNX — a cross-framework portability format; less common for LLMs specifically than safetensors/GGUF.

A generative model (the subject of this whole guide) produces text token-by-token. An embedding model instead converts text into a fixed-length vector representing its meaning, used for search/retrieval (semantic search, RAG). Different model, different use case.

Most local inference tools expose an HTTP API mimicking OpenAI's /v1/chat/completions shape. Any code written against OpenAI's API can point at http://localhost:PORT/v1 instead and just work — no code changes beyond the base URL. This is the single most useful interoperability convention in this ecosystem; every tutorial in this guide relies on it.