Ollama

The lowest-friction way to get a model running locally.

Install

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# or on macOS: download the .app from https://ollama.com/download

Pull and run a model

ollama pull qwen3:8b     # downloads the GGUF-packaged model + tokenizer
ollama run qwen3:8b      # drops you into an interactive chat prompt

The model tag (qwen3:8b) follows name:size convention; Ollama pulls a pre-quantized GGUF automatically (Q4_K_M by default). Type /bye to exit.

Use it as an API from code

Ollama runs a local server on port 11434 with an OpenAI-compatible endpoint:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [{"role": "user", "content": "Write a haiku about pointers."}]
  }'

From Python, using the official OpenAI SDK pointed at your local server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # api_key is unused but required by the SDK

response = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Explain quicksort in one paragraph."}],
)
print(response.choices[0].message.content)

Customize behavior with a Modelfile

A Modelfile (analogous to a Dockerfile) lets you set a system prompt, default parameters, or start from a different base:

# Modelfile
FROM qwen3:8b
PARAMETER temperature 0.3
SYSTEM "You are a terse senior code reviewer. No pleasantries."

ollama create code-reviewer -f Modelfile
ollama run code-reviewer

Useful commands

ollama list          # models you have locally
ollama ps             # models currently loaded in memory
ollama rm qwen3:8b    # delete a model
ollama show qwen3:8b  # metadata: quantization, context length, etc.