A few weeks ago I asked myself whether I could make my household inventory agent work without the Anthropic API. Not because I'm unhappy with Claude — I'm not — but because "can I run this locally?" is a useful question to answer for any system I build professionally. Some clients have strict data residency requirements. Others just don't want to depend on an external API.
The answer is: yes, but with meaningful tradeoffs. Here's what I found.
The setup
Ollama is the easiest way to run local models on a Mac or Linux machine. You install it, pull a model, and get an OpenAI-compatible REST API at localhost:11434. The OpenAI-compatible part is key — it means you can swap it in for any code that uses the OpenAI SDK, with almost no changes.
# Install Ollama, then:
ollama pull llama3.1 # general purpose, tool use
ollama pull qwen2.5-vl # multimodal (vision)
# API is at: http://localhost:11434/v1The code change to swap from Claude to Ollama is basically:
# Before (Anthropic SDK)
import anthropic
client = anthropic.Anthropic(api_key=settings.anthropic_api_key)
# After (OpenAI SDK pointed at Ollama)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")You also need to convert tool definitions from Anthropic's format to OpenAI's function format — they're structurally similar but not identical. And the tool result format in the message history is slightly different. Plan an hour or two for that migration.
Where local models shine
Simple, well-defined tasks. Extracting structured data from text, summarizing a document, answering questions with context provided in the prompt — local models handle these well. A llama3.1:8b can do most of these tasks fast and with quality that's good enough for many use cases.
High-volume, low-stakes inference. If you're running thousands of inferences per day on short prompts, the cost math often favors a local model after you account for the GPU or Apple Silicon compute. The break-even depends heavily on the task and your hardware.
Data privacy requirements. If you're processing sensitive customer data that can't leave your infrastructure, local models are often the only viable path. Self-hosted models with Ollama + Docker make this surprisingly manageable.
Latency-sensitive applications. With the right hardware (an M2 Pro or better, or a machine with a decent GPU), local inference is faster than a round trip to a cloud API for small models.
Where I ran into trouble
Multi-step tool use. This is where the gap between local models and frontier models is most visible. My inventory agent uses a loop of up to 20 tool calls per turn — checking categories, adding items, looking up existing entries. Claude handles this reliably. With llama3.1:8b, the agent started producing malformed tool calls around step 5-6 and sometimes hallucinated tool names that didn't exist in the schema. qwen2.5:14b was better, but still not as reliable.
Long context. Local models still struggle with truly long contexts. At 32k+ tokens, I started seeing the model lose track of earlier tool results. Claude Sonnet handles 200k tokens. That gap matters for agentic loops with lots of tool call history.
Vision quality. llava and qwen2.5-vl are genuinely impressive for local multimodal models, but they're noticeably weaker than Claude at recognizing partially obscured items, reading small text on packaging, or handling poor lighting. My pantry recognition endpoint would miss 2-3 items per photo compared to Claude's output.
Following complex instructions. My system prompt is detailed — it tells the agent to check categories before adding items, prefer certain tool call orderings, and ask for clarification when input is ambiguous. Frontier models follow these instructions reliably. Smaller local models follow them sometimes.
My recommendation
Use local models when:
- The task is well-defined and doesn't require multi-step reasoning
- Data privacy prevents sending data to cloud APIs
- You're comfortable with somewhat lower accuracy
- You have a modern Mac with 16GB+ RAM or a machine with a decent GPU
Stick with cloud APIs when:
- Your application depends on reliable multi-step tool use
- Vision accuracy matters
- You're iterating fast and don't want to manage infrastructure
- The reliability ceiling of local models isn't high enough for your error tolerance
For most client work I do at Solvbit, cloud APIs are the right answer. But for prototyping, high-volume pipelines, and privacy-sensitive deployments, local models are now a real option — and they're improving fast.
If you're curious about the exact code changes needed to swap Ollama into a FastAPI / tool-use setup, shoot me a message — happy to share the diff.