Building a Household AI Agent with Claude Tool Use

I've been building a household inventory manager as a side project. The premise is simple: take a photo of your pantry, and the system figures out what's there, what's running low, and what you could make for dinner. Simple enough as an idea. Surprisingly instructive as a build.

The interesting part isn't the inventory database or the mobile app. It's the agent in the middle — a Claude-powered reasoning loop that has tools for seeing, looking things up, and updating records. Getting that loop right took longer than I expected.

The architecture

The stack is: FastAPI backend, PostgreSQL for storage, Expo / React Native for the mobile app, and the Anthropic Claude API for the agent and vision. Everything runs in Docker.

The agent is a standard tool-use loop: call Claude with a message and a list of tools, check if Claude wants to call a tool, execute it, feed the result back, repeat until Claude gives a final answer. The version I have runs for up to 20 iterations with a 4,096-token output budget per turn.

The tools the agent has access to:

• get_inventory — list all items with current stock levels
• get_low_stock — items below a set threshold
• adjust_stock — decrement or increment quantity
• add_item — create a new item with initial stock
• update_item — change an item's name, category, or unit
• list_categories / list_locations — context for organizing items
• create_category — if a needed category doesn't exist

Photo recognition is handled separately. When you send a photo, it goes to a dedicated /agent/recognize-photo endpoint that calls Claude Vision with a base64-encoded image and asks it to return structured JSON — a list of items with name, estimated quantity, unit, and suggested category. That list gets formatted as a text message and dropped into the agent chat, where the agent can then call add_item in a loop to create them all.

What I got wrong first

My first version had 4 tools and a 10-iteration limit. It fell apart on anything that required more than two steps. "Add a dozen eggs and some olive oil" would sometimes add the eggs but forget the oil. The agent was hitting the iteration cap before finishing.

Bumping to 20 iterations and 4,096 tokens helped — but more importantly, I rewrote the system prompt. The original was too sparse. Claude is good at reasoning, but it still needs clear instructions about when to use which tool and in what order.

The rewrite told the agent to:

1. Call list_categories first when adding new items, so it can assign proper categories rather than guessing
2. Check existing inventory before creating new items to avoid duplicates
3. Confirm quantities and units with the user if the input is ambiguous

That alone cut most of the failure modes.

The vision part is easier than you'd think

I was bracing for vision to be the hard part. It wasn't. Claude's vision is genuinely good at identifying packaged food items. You send a base64-encoded image inside a message content block, ask for structured JSON, and it reliably comes back with a list of recognizable items.

content = [
    {
        "type": "image",
        "source": {
            "type": "base64",
            "media_type": "image/jpeg",
            "data": base64_image,
        },
    },
    {
        "type": "text",
        "text": "Identify all visible pantry/food items. Return JSON array: [{name, quantity, unit, suggested_category}]"
    }
]

The bigger challenge was mobile camera quality and framing. A well-lit photo of organized shelves is identified almost perfectly. A dark, cluttered photo of a messy pantry is much harder. I ended up setting the mobile app to compress images to 70% quality (smaller payload, faster) and prompt the user to aim the camera at one shelf at a time.

What I'd build differently

A few things I'd change if I were starting over:

Separate the read and write tools more clearly. Right now the agent can call get_inventory and add_item in the same turn. That's fine, but it means I need careful prompt guidance to prevent unnecessary reads. A cleaner design might split the agent into two phases: a planning phase (read-only tools) and an execution phase (write tools).

Add a confirmation step for bulk adds. When recognizing from a photo, the agent might try to add 12 items at once. If one call fails mid-way (network issue, duplicate item), the state ends up inconsistent. I should buffer the add requests and execute them as a transaction.

Evaluate the system prompt properly. I've been tuning the prompt by hand, which works but doesn't scale. I want to build a small eval suite — a set of example user messages and expected tool call sequences — and run it every time I change the prompt.

Is the project useful?

Genuinely, yes. I've been using it for a few months. The thing I use most is the low stock alert — I've stopped running out of things I go through regularly. The photo recognition is a party trick that's occasionally useful for a big restock after grocery shopping.

More than the app itself, though, building it has been a good test of how agentic systems behave in practice. The failure modes are predictable once you've seen them: ambiguous instructions, running out of context, tools that return inconsistent data. None of them are hard to fix if you're watching for them.

The code is on GitHub if you want to dig in.