An agent is just a loop. The model decides which tool to call, your code calls the tool, the result goes back to the model, repeat. The reason agents fail in production is not the model — it’s everything around the loop.
We’ve shipped four agent systems in the last year. Three of them work well. The one that didn’t taught us most of what’s in this post.
Tool schemas are a contract, not a hint
The model will only call the tools you describe. The descriptions matter as much as the parameters. If your send_email tool’s description says “sends an email,” the model will use it for every email-shaped intent — including drafts, reminders, and notifications you didn’t intend it to handle. If the description says “sends an email to a customer; only call after explicit user confirmation,” the failure mode shifts.
Keep parameter schemas tight. Use enums where you can. Mark required fields. Validate everything that comes back from the model before passing it to the actual tool — every production agent we’ve shipped has had at least one bug where the model invented a field that looked plausible but didn’t exist.
The retry policy decides the user experience
Network errors, rate limits, transient model failures — these all happen. The naive answer is to retry. The right answer depends on whether the tool was idempotent.
We split tools into two buckets. Read tools (search, lookup, fetch) get retried automatically with exponential backoff. Write tools (send, create, update, delete) do not get automatic retries. They surface the failure to the model, which then decides whether to call again, ask the user, or escalate. This avoids the failure mode where a flaky network sends three identical purchase orders.
Humans in the loop, not on the bench
Every agent we’ve shipped has at least one tool that requires explicit human approval. The pattern is the same: the model proposes the action, the system serializes it to a structured payload, a human sees the payload in a UI and clicks approve or reject, the result goes back to the model.
The trick is making the approval UI fast enough that humans actually use it. If approval takes thirty seconds because the user has to dig through context, they’ll start rubber-stamping. We render the agent’s reasoning trace next to the proposed action, so the human can audit the why, not just the what.
Three production examples
The sales-ops agent for a B2B distributor reads incoming RFQs, drafts quotes against a 60k-SKU catalog, and queues them for human approval. Read tools are autoretry. The drafted quote is the unit of approval. A human reviews and clicks send. Throughput went from 40 quotes a day to 600.
The support triage agent for a SaaS company reads incoming tickets, looks up the customer, classifies the issue against a known taxonomy, and proposes an action. Some actions (close as duplicate, request more info) execute automatically. Some (refund, account suspension) require human approval. The agent’s accuracy on classification is 91%; the auto-execute actions have a one-click “undo” for the support team.
The documentation agent for an internal engineering team reads pull requests, looks up affected code paths, and drafts release notes. There is no auto-execute path. Every output is reviewed. The win is not eliminating the human — it’s eliminating the empty-page problem at 5pm on a Friday.
The one that failed
We built an agent that managed deployment rollouts based on metric signals. It worked in staging. In production, it kept making correct decisions for the wrong reasons, then occasionally making confidently incorrect decisions when a metric source went stale. The retry policy compounded the errors. We pulled it after two weeks.
The lesson: agents are not appropriate for actions where the cost of an incorrect call is severe and the recovery cost is high. Use them for high-volume, low-stakes work. Use them for drafting. Use them for triage. Don’t use them for anything you couldn’t undo with a rollback button.