AI agents in production: let code measure and the model judge
What actually breaks when an LLM agent leaves the demo and meets real users, and the architecture rule that fixes most of it.
Agents are the most important shift in software this year. Instead of answering one prompt, a model now plans a task, calls tools, reads the results and decides what to do next. The demos are impressive. Production is a different story, and most of the trouble has nothing to do with how clever the model is.
I build agents that run every day for real businesses: an SEO teardown agent that audits law-firm websites, a job-discovery pipeline, a multi-agent SEO monitoring system and a local voice assistant. Each one broke in production in a way the demo never showed. The fixes all pointed to the same rule.
The rule: code measures, the model judges
Language models are excellent at reading messy input, classifying it and writing clear text. They are unreliable at producing the same number twice. So I split every system along that line:
- Deterministic code crawls pages, counts things, runs checks and computes every score.
- The model plans which checks to run, interprets what they found and writes the explanation.
When an early version of my SEO agent told a firm it had no attorney biographies, the firm had 47 of them. The model had sampled a handful of pages and drawn a confident conclusion. The fix was not a better prompt. I split "does it exist" checks, where every URL is classified by code, from "how good is it" checks, where the model reads a sample. The scoring itself moved entirely into code that the model cannot nudge.
What breaks first
1. Quotas and rate limits
One agent run can make 15 to 20 model calls. On a free or low tier, the second run of the day dies halfway through. Treat the model like any other rate-limited API: a shared token bucket across all workers, exponential backoff on 429 responses, and a cheaper model alias for the steps that do not need the strongest one.
2. Models disappear
Providers retire model versions, sometimes only for new API keys. If the model name is hard-coded in five places, you find out from a user. Keep the model name in one config value, version your prompts alongside it, and have a fallback model ready.
3. Upstream data changes silently
My rank-tracking tool once reported that every site had dropped out of Google. Nothing had dropped. Google had changed its result links to opaque redirects, and the parser quietly found nothing. Now a run with zero organic results raises a hard error. If the outside world changes, the tool fails loudly instead of lying quietly.
4. Runaway tool calls and invented tools
Agents sometimes call tools that do not exist or loop on the same call. Validate every tool name against the real spec, cap the number of steps per run, and fall back to a plain chat answer when the plan goes off the rails.
Guard rails that pay for themselves
- Timeouts with a null fallback. If the model is slow, the report ships without the advisory paragraph instead of blocking.
- Structured output, validated. Parse the model's JSON against a schema and reject anything that does not fit.
- Bounded writing. For anything sent to a person, such as a cover letter or an outreach email, use a template and let the model fill specific slots. It cannot invent a job history it was never given.
- Binary questions instead of free scores. A small local model gave a perfect-match job 20 out of 100. Five yes-or-no questions mapped to fixed points gave identical results across repeated trials.
- Dry-run by default, daily caps and a kill switch for anything that acts in the outside world.
The takeaway
The teams getting real value from agents are not the ones with the cleverest prompts. They are the ones who decided early which parts of the system must be exact and kept the model away from those parts. Let the model do what it is good at, judgement and language, and let code do the rest.