Production AI agents, tested before they run unattended.
An AI agent is easy to demo and hard to trust. The model is the small part. Most of the work is what surrounds it: an evaluation set, tracing, approval gates and an audit trail, so the agent can run every day without someone watching it.
What is included.
- An evaluation set built from your own real cases before anything ships. It reruns before every prompt or tool change.
- Failures sorted by type, so we fix causes instead of symptoms.
- Tracing on every request for cost, latency and tokens.
- Approval gates on any write to a system of record. When the agent isn’t sure, a person decides.
- Tool definitions and schemas under version control, reviewed like any other code.
- A full audit trail, because someone will make a real decision on the output.
What clients use it for.
Revenue reconciliation agent
Reconciles customer and billing records across six systems every day, and sends the cases it cannot resolve to a person.
In production: National wireless carrier →AI agent evaluation
Builds an evaluation set from your own cases and reruns it before any prompt, tool or model change ships, including on agents you already run.
How it works.
Most agent failures are not prompt failures.
On the telecom reconciliation agent, the evaluation set showed that most failures came from how the tools were defined. Two tools described overlapping jobs, or a schema allowed a parameter the API would reject. The model did what the definitions told it to do. We rewrote the tool contracts and put them under version control, so a change to a tool is reviewed and tested like any other code change.
Evaluation comes before autonomy.
We collect real cases from production traffic, write down the correct outcome for each, and turn them into a regression suite. Every change to a prompt, a tool or a model reruns the suite before it ships. When something fails, we record the failure type, so we fix the cause of a pattern instead of patching single cases.
A person decides the expensive calls.
The agent reads freely and writes carefully. Any write to a system of record passes an approval gate, and cases the agent is unsure about go to a person with the context they need to decide. Every action, approved or not, lands in an audit trail.
Claude, OpenAI and AWS Bedrock.
We build on the Claude and OpenAI APIs and on AWS Bedrock, with MCP for tool access, and choose the model for each task. Obito Digital is an OpenAI Select Partner.
Keep reading.
Questions about the production AI agent platform.
Not covered here? Send us your question and an engineer will answer it.
Ask a question- Why do AI agents fail in production?
- In our experience, mostly because of how the tools are defined. Overlapping tool descriptions and schemas that allow parameters the API will reject cause more failures than the prompt or the model.
- How do you test an AI agent before it runs unattended?
- We build an evaluation set from your own real cases, sort failures by type, and rerun the set before every prompt or tool change.
- What happens when the agent is not sure?
- A person decides. Any write to a system of record goes through an approval gate, and every action is recorded in an audit trail.
- Do you build on Claude or OpenAI?
- Both. We build on the Claude and OpenAI APIs and on AWS Bedrock, and choose the model for each task. Obito Digital is an OpenAI Select Partner.
- What does it cost to build a production AI agent?
- The one-week audit is a flat fee, set by your industry and company size. Setting the agent up on your systems is a fixed fee agreed after the audit. After that there are two monthly costs: the platform license, which covers running and maintaining it, and a fixed fee for the team.
- How long until an agent runs in production?
- Discovery and audit take one week. After the audit, the platform runs on your real data within 90 days at most, and a proof of concept can be ready in as little as 24 hours.