Agent frameworks and local models

What you’ll learn

What agent frameworks do for you, how the main options differ, how to point them at a local model, and a realistic sense of how well local agents perform.

What a framework gives you

You can write an agent loop yourself in not many lines of code. A framework earns its place when you need more than the basics: managing state across steps, coordinating several agents, persistence and recovery, human-in-the-loop approval, and observability. It handles the plumbing so you can focus on the task.

The main open-source options suit different needs:

• LangGraph models an agent as a graph with explicit state, branching, and human-in-the-loop steps. It is the most battle-tested for complex, stateful systems that need audit trails and recovery points.
• CrewAI uses an intuitive role-and-crew paradigm, and is one of the quickest ways to stand up a multi-agent prototype.
• Letta specialises in long-term memory, for agents that need to remember across sessions.
• Pydantic AI brings type safety and structured outputs, a good fit when an agent’s results feed into other code.

All of them are model-agnostic, which is what lets them run against local models.

Running against a local model

The connection is simpler than it sounds. Serve a tool-capable model through Ollama, which exposes an OpenAI-compatible endpoint, then point the framework at that endpoint instead of a commercial API. From the framework’s perspective it is talking to an OpenAI-style API; it does not care that the model is running on your own machine.

Choose a model suited to agentic work. The Qwen3 30B-A3B mixture-of-experts model is a strong local choice, capable at tool use and quick thanks to its design; Qwen3 8B is a lighter option for simpler agents. Both benefit from as much context as your hardware allows.

A realistic view

It is worth setting expectations honestly. Frameworks make agents easier to build, but they do not make a local model more capable than it is. Multi-step, multi-agent work is demanding, and local models at consumer sizes are less reliable at it than the best cloud models. The more agents and steps you add, the more chances there are for one to go wrong and derail the rest.

That is not a reason to avoid local agents. It is a reason to start small: one agent, few tools, a narrow task, with supervision. Prove that works before adding complexity. Many capable local agent setups are deliberately simple, and simplicity is often what makes them reliable.

What can go wrong

• Reaching for multi-agent orchestration too soon. More agents means more failure points. A single well-scoped agent is more reliable, and often enough.
• Blaming the framework for the model. If an agent reasons poorly, a different framework will not fix it. Try a stronger or more tool-tuned model first.
• Skipping limits and oversight. Set step limits and timeouts, and keep a human in the loop for anything consequential, exactly as with coding agents.

Next steps

For agents aimed specifically at software work, see local coding agents. The agent frameworks in the catalogue cover each option and its trade-offs.