Running AI Models Locally Without the Cloud
Ollama makes it genuinely straightforward to run large language models on your own hardware – no API keys, no usage limits, no data leaving your machine. If you have a modern CPU or a supported GPU, you can have a working local AI server running in under ten minutes.
What Ollama Is and Why It Matters
Ollama is an open-source tool that packages model management, a local inference server, and a clean command-line interface into a single installable binary. It handles the heavy lifting of downloading quantized models, managing GPU layers, and exposing a REST API that mimics the OpenAI format – which means most tools built for ChatGPT’s API can point at Ollama instead with minimal reconfiguration.
The models Ollama supports span a wide range: Meta’s Llama 3, Mistral, Google’s Gemma, Microsoft’s Phi, DeepSeek, Qwen, and a growing library of fine-tuned variants. Each model is stored in Ollama’s own format, versioned and cached locally. You pull a model once, and it stays on disk until you remove it. No re-downloading on each session, no cloud dependency mid-conversation.
Hardware requirements vary sharply by model size. A 7B parameter model in 4-bit quantization typically fits in 4-6 GB of VRAM – or runs on CPU with 8 GB of system RAM, though CPU inference is noticeably slower. A 13B model wants 8-10 GB of VRAM. If you are running an NVIDIA RTX 3060 or better, most 7B and 13B models will run at a comfortable speed. Apple Silicon Macs handle this particularly well because of unified memory: a MacBook Pro with 16 GB of RAM can run 13B models with reasonable response times.
One clarification worth making upfront: Ollama is a server and model manager, not a chat interface. It exposes endpoints you query directly via curl, or through front-end apps built on top of it. That separation is intentional and useful – it means you can swap interfaces without changing anything about how models are configured or served.
Installing Ollama and Pulling Your First Model
Installation differs by operating system, but each path is short. On Linux, the official install script handles everything:
curl -fsSL https://ollama.com/install.sh | sh
That command downloads the binary, installs a systemd service, and starts Ollama automatically. On macOS, download the .app from ollama.com and drop it into Applications – it runs as a menu bar app and starts the server in the background. On Windows, Ollama is available as a standard installer from the same site, and it runs as a background process after install. GPU support on Linux requires the appropriate NVIDIA or AMD drivers already installed; Ollama detects them automatically and falls back to CPU if they are absent.
Once Ollama is running, open a terminal and pull a model. For a first test, Llama 3.2 in its 3B variant is fast and light:
ollama pull llama3.2
Ollama will download the quantized model file, which is around 2 GB for the 3B version. Larger models like llama3.1:8b sit around 4.7 GB. After the pull completes, run an interactive session directly from the terminal:
ollama run llama3.2
That drops you into a prompt where you can type directly and get responses. It is useful for quick testing, but the more practical use case is the API. Ollama starts an HTTP server on port 11434 by default. You can verify it is running by hitting http://localhost:11434 in a browser – you will see a plain “Ollama is running” response. From there, queries follow the OpenAI chat completions format:
curl http://localhost:11434/api/chat -d ‘{“model”: “llama3.2”, “messages”: [{“role”: “user”, “content”: “Explain DNS over HTTPS in two sentences.”}]}’
Responses stream back by default. Add “stream”: false to the JSON body if you want a single complete response instead. The API also supports /api/generate for raw completion without a message history, and /api/embeddings for generating text embeddings – useful if you are building retrieval-augmented generation pipelines locally.
Adding a Chat Interface and Configuring Ollama as a Service
For most people, querying a local API via curl gets old quickly. Open WebUI is the most widely used front-end for Ollama – it is a self-hostable web app that provides a chat interface nearly identical to ChatGPT, with conversation history, model switching, and system prompt configuration. Deploy it with Docker in a single command:
docker run -d -p 3000:8080 –add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data –name open-webui –restart always ghcr.io/open-webui/open-webui:main
After that, Open WebUI is available at http://localhost:3000. On first launch it asks you to create an admin account, then auto-detects any Ollama models you have pulled. Switching models mid-conversation is a dropdown. You can define system prompts per conversation, and the interface stores full chat history locally in a SQLite database. Nothing touches the internet unless you explicitly enable optional integrations.
If you want Ollama accessible from other devices on your local network – not just localhost – set an environment variable before or alongside starting the service. On Linux with systemd, edit the service file at /etc/systemd/system/ollama.service and add Environment=”OLLAMA_HOST=0.0.0.0″ under the [Service] block, then reload and restart with systemctl daemon-reload && systemctl restart ollama. On Mac, you can set it via the terminal before launching. Once bound to 0.0.0.0, any device on your network can reach the API at your machine’s local IP on port 11434 – which also means Open WebUI running on another machine can point at it. Keep that access internal to your network; exposing Ollama to the public internet without authentication is not something you want to do casually, since the API has no built-in auth layer. If you are already running local network services and managing DNS internally, the same network hygiene that applies to something like Pi-hole with Unbound applies here – keep sensitive local services off the public-facing side of your router.
Model management over time is worth thinking through. Ollama stores models in ~/.ollama/models on Linux and Mac, and each model can run 2-8 GB per file depending on the variant. Use ollama list to see what is installed and ollama rm modelname to free space. You can also create custom Modelfiles – Ollama’s equivalent of a Dockerfile for models – to bake in system prompts, adjust context length, or set a specific temperature by default. A Modelfile for a coding assistant with a fixed system prompt takes about five lines and means you never have to paste the same prompt again.
