Self-hosted AI agents: what to run and on what

What you’ll learn

Which agent platforms you can run on your own server, what each is good for, and whether each one can drive a local model or only a cloud API. Plus the two things the marketing tends to skip: how reliable local models actually are at agentic tool use today, and what you are taking on by letting an agent act on your machine.

“Self-hosted” is not the same as “local model”

This is the distinction to get straight first. A self-hosted agent platform runs on your own infrastructure, but many such platforms still send the actual thinking to a cloud model by default. Running the orchestration yourself keeps the workflow and your data on your hardware, but if the model is a hosted API, your prompts still leave the building. To be fully local, you point the platform at a local model endpoint, typically something serving an OpenAI-compatible API such as Ollama or vLLM. Every tool below can work with a local model; check that you have actually configured it to, rather than assuming self-hosted means local.

The tools you can run yourself

These are the agent tools in our solutions catalogue that run on your own hardware and work with local models. Each suits a different job.

Tool	Best for	Works with local models	Licence
OpenHands	Autonomous coding, feature-sized tasks in a sandbox	Yes	MIT
CrewAI	Multi-agent workflows with role-based agents	Yes, one of the easiest to point locally	MIT
Letta	Agents that need long-term memory across sessions	Yes	Apache 2.0
Dify	Visually building agentic workflows, RAG, and LLM apps	Yes, multi-model	Dify Open Source License

A rough guide: reach for OpenHands if you want an agent that writes and runs code to complete a task; CrewAI if you are assembling several cooperating agents and want the gentlest start with a local model; Letta if persistent memory is the point; and Dify if you want a visual builder for workflows and RAG rather than code. Licences differ, so read each before commercial use, particularly Dify’s own licence, which is not a standard open-source one.

How far local models actually get with agents

Honestly: further than a year ago, but not as far as the demos suggest. Agentic work leans hard on reliable tool use, calling functions with the right arguments, over many steps without drifting, and this is exactly where local models still trail the hosted frontier. The practical consequences:

• Reliability drops with complexity. A local model may handle a two or three step task well and then lose the thread on a long one. Expect to supervise.
• Model choice matters more than framework choice. A capable, tool-use-trained model is worth more than a clever harness. See choosing a local coding model and local coding agents for the coding case, which is where local agents are strongest today.
• Context and speed bite. Agents generate a lot of tokens, so a model that is merely usable in chat can feel painfully slow in an agent loop, and long tasks can overrun a short context window.

None of this means local agents are not worth running. It means setting expectations: they are genuinely useful for scoped, supervised tasks, and improving quickly, rather than a drop-in replacement for a frontier hosted agent on hard work.

Letting an agent act on your machine

An agent that can run commands, edit files, or call tools is powerful precisely because it acts, and that is also the risk. A few principles worth holding:

• Sandbox it. Prefer tools that isolate execution; OpenHands, for example, runs each task in a Docker container rather than directly on your system.
• Least privilege. Give it access to the files, directories, and credentials the task needs and no more. Do not hand an agent broad access to secrets or your whole home directory out of convenience.
• Watch the network boundary. If you expose an agent or its model API beyond localhost, you have opened a door; only do so on a network you trust.
• Keep a human in the loop for anything irreversible. Review actions that delete, send, publish, or spend before the agent takes them.

Our guide to tool use and MCP covers how agents reach the outside world, which is the same surface you are securing here.

Which should you choose?

• Coding tasks: OpenHands, with a capable local coding model and its Docker sandbox left on.
• Several agents cooperating, easiest local start: CrewAI.
• Memory across sessions: Letta.
• A visual builder for workflows and RAG: Dify.

Whichever you pick, decide deliberately whether the model is local or a cloud API, and start with a small, supervised task before trusting it with a larger one.

Next steps

• The foundations: what is an agent and agent frameworks and local models.
• How agents reach tools: tool use and MCP.
• Serving a local model for an agent to use: Ollama or vLLM.