How to run an LLM locally: your first model

What you’ll learn

How to go from nothing to a working local model you can chat with, and then call from your own code. We use Ollama because it is the fastest route for a first run; if you would rather not use a command line at all, skip to the no-command-line route near the end.

First, will it run?

Running a model locally is mostly a question of memory. A model has to fit in your RAM, or better your GPU’s VRAM, with room to spare for the conversation. Not every machine runs every model, so start small and work up rather than reaching for the biggest model first.

As a rough guide, the small models in this guide need only a few gigabytes and run on most recent laptops, including on the CPU if you have no dedicated GPU. Before you pick anything larger, check what your hardware can actually handle on the hardware matrix, and read how much memory do I need for the fuller picture. Remember that “fits” is not the same as “runs quickly”.

Install Ollama

Ollama is the inference engine and model manager we will use. Install it for your system:

• macOS: download the app from ollama.com, open it, and follow the prompt. This installs the ollama command.
• Windows: download the installer from ollama.com and run it.
• Linux: run the official install script:

  curl -fsSL https://ollama.com/install.sh | sh

You can confirm it is installed by opening a terminal and running ollama --version.

Pull and run your first model

Now run a small model. Llama 3.2 3B is a good first choice: it is small (a download of roughly 2GB) and runs comfortably on modest hardware.

ollama run llama3.2:3b

The first time, Ollama downloads the model; after that it starts straight away. When it is ready you get a prompt where you can type a message and get a reply. Type /bye to leave the chat.

If you have around 8GB of memory free and want a more capable model, try Qwen3 8B instead:

ollama run qwen3:8b

That is local AI working: the model is running on your machine, and nothing you type is leaving it.

Talk to it from your own code

Ollama also runs a local server, on port 11434, so your own scripts and tools can use the model through an API. With Ollama running, this returns a completion:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Explain what an LLM is in one sentence.",
  "stream": false
}'

The server also exposes an OpenAI-compatible endpoint at http://localhost:11434/v1/chat/completions, which means a great many existing tools and libraries can point at your local model by changing only the base URL. That is why Ollama is a common first choice: the model you ran a moment ago is already an API other software can use.

If it is slow, or will not fit

If a model is sluggish, or Ollama reports it cannot load, the usual cause is memory. A few things to try, in order:

• Use a smaller model. Drop from an 8B to a 3B, or smaller still. A model that fits with headroom will always feel better than one that barely fits.
• Use a more compressed version. Models come in different quantisations, and a lower-bit build needs less memory for a modest quality cost. See understanding quantisation.
• Shorten the conversation. A long context uses more memory as it grows, so starting a fresh chat can help.
• Check the numbers. The hardware matrix shows which models fit your memory and at what quantisation, which saves trial and error.

Prefer not to use a command line?

If the terminal is not for you, LM Studio does the same job through a graphical app: you browse for a model, click to download, and chat in a window, with the same kind of local server underneath. Our LM Studio vs Ollama comparison covers which suits you.

Next steps

• Pick a model deliberately for what you want to do: choosing your first model.
• Understand the tool you used, and its alternatives: inference engines explained.
• See exactly what your hardware can run: the hardware matrix.