Hands-On Tutorial 4

vLLM for high-throughput serving

Use this once "local inference" becomes "a service other people or processes hit concurrently" — a team-internal tool, a multi-user app, an agent swarm.

vLLM's key innovation is PagedAttention, which manages the KV cache like an OS manages virtual memory pages, letting it batch many concurrent requests far more efficiently than naive serving.

Install and run

Requires an NVIDIA GPU on Linux.

pip install vllm --break-system-packages

vllm serve Qwen/Qwen3-8B-Instruct \
  --port 8000 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192

Or via Docker, without a local Python environment at all:

docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B-Instruct

This also exposes an OpenAI-compatible endpoint at http://localhost:8000/v1 — same client code as the Ollama example in Tutorial 1 works here unchanged, just point at a different port. See the vLLM Online Serving docs.

Key flags worth knowing

--tensor-parallel-size N — splits a model across N GPUs
--max-num-seqs — caps concurrent request batching
--dtype — forces a specific precision