What is local AI?
Running AI models on your own hardware instead of calling a cloud API. What that means in practice, what it is genuinely good for, and where cloud is still the better choice.
What you’ll learn
What “local AI” actually means, why people run models on their own machines, and how to tell when it is the right choice and when it is not.
The concept
Local AI means running a model on hardware you control, a laptop, a desktop with a graphics card, or a small server, rather than sending your requests to a provider’s API. The model’s weights sit on your disk, load into memory, and generate responses without anything leaving your machine.
People choose it for a few reasons. Privacy is a common one: your data never goes to a third party, which matters for sensitive documents, regulated work, or simply not wanting your inputs logged elsewhere. Others value not depending on a service that can change its pricing, terms, or model behaviour. And some do it because it is genuinely interesting to run this technology yourself.
It is worth being clear about the trade-offs from the start. Local models at consumer scale still lag the best cloud models on hard tasks. Setup takes more effort. Good hardware is expensive. And running a model yourself is often slower than a well-provisioned API. None of this makes local AI a bad idea. It makes it a considered one.
When cloud is the better answer
If you want the most capable model available with no setup, a cloud API is usually the right choice, and we will say so throughout this site. Local AI earns its place when privacy, control, cost at scale, or learning matters to you, not as a default that is always superior. A reader who tries local AI, finds it does not suit their needs, and goes back to an API has still been well served.
Doing it
The gentlest way in is a small model and a tool that manages the details for you. Ollama will download and run a model with a single command, and a small model like Llama 3.2 3B runs on modest hardware, including laptops without a dedicated graphics card. That combination is enough to see what local AI feels like before you spend anything.
What can go wrong
- Expecting frontier quality from a small model. A 3B model is useful, but it is not going to match a large cloud model. Judge it as what it is.
- Running out of memory. Models need to fit in your available memory. If a model is too large, it will either refuse to load or fall back to painfully slow behaviour. Our hardware matrix helps you see what fits.
Next steps
Once you have a model running, the next thing worth understanding is quantisation, which is what makes running these models on ordinary hardware possible in the first place.