The demo went perfectly. The agent read a customer email, checked the order system, drafted a refund, and cc'd the right manager, live, in front of the executive team, on the first try. Someone actually applauded. Six weeks later the same agent, now connected to the real inbox, promised a customer a full refund, a replacement unit, and, for reasons nobody has fully reconstructed, a handwritten apology from the CEO. The customer was delighted. Legal was not.
How to deploy AI agents in production, the honest summary: treat the demo as evidence of possibility, not readiness. Map the human workflow first, build evals from real cases before you tune anything, ship guardrails as a product feature, roll out in three phases, and monitor forever. The model is the easy part. Everything around the model is the job.
This is the playbook FDEs use when a cool demo needs to become infrastructure. It is not glamorous. It works anyway.
The Demo-to-Production Chasm
Across embedded AI engagements the pattern is consistent, and the numbers that follow are illustrative but directionally honest: of every 100 agent demos, maybe 55 survive the process of building a real evaluation set, about 30 make it through a serious guardrail review, and around 15 end up in daily production use. The other 85 didn't fail because the models were bad. They failed because demos run on happy paths and production runs on your worst Tuesday.
Your worst Tuesday includes the customer email that's a photo of a PDF of a screenshot, the API that returns null in a field that's been a string for three years, and the edge case your top performer handles by instinct and never documented. Productionizing LLM agents means engineering for that Tuesday, which is why the gap between pilot and production swallows most projects whole.
Start With the Workflow, Not the Model
Before anyone touches a prompt, map the human workflow the agent is joining. Then sort every step into three buckets: steps the agent owns outright, steps where it drafts and a human approves, and steps it must never touch. That third bucket matters more than the other two combined.
The most useful rule here is the dollar threshold. An agent that can draft a response to anything but only execute actions under, say, $500 without human approval is an agent you can actually deploy. The threshold isn't a technical parameter. It's a negotiated agreement with the business about how much trust has been earned, and it should move only with evidence.
Agent vs. boring script: an honest fork in the road
Some of your workflow steps are deterministic. If the rule is "invoices over $10k from new vendors get flagged," that's a script, and writing it as an agent means paying tokens for coin flips you didn't need to take. Agents earn their keep on steps with genuine ambiguity: unstructured text, judgment calls, novel inputs. Everything else gets code. The boring script has an uptime the agent vendors don't put in the brochure.
Evals Before Elegance
The single most valuable artifact in any agent project is a test set of real cases, built before anyone tunes anything. Pull 50 actual examples from the workflow: the easy ones, the ugly ones, and the ones that made a human stop and think. Score the agent against all of them. That number, not the demo, is your baseline.
From then on, every change gets measured against the set. Prompt tweak, model swap, new tool, all of it. Regression-catching becomes a weekly habit rather than a quarterly heart attack, and arguments about whether a change "feels better" get replaced by a number that settles the discussion in the time it takes to run the suite. The full method is in our guide to evals for LLM features.
Guardrails Are a Product Feature
Guardrails get treated as compliance paperwork when they should be treated as the thing that makes the agent shippable. The working set: output validation that catches malformed or policy-violating responses before they leave the building, tool-permission scoping so the agent can only call the four APIs it needs rather than the forty it can see, rate limits so a loop becomes an alert instead of a billing event, and a defined escalation path to a human with the context attached.
Remember our refund-promising friend from the intro? Postmortem findings: the agent had permission to send email, a generous interpretation of "make the customer happy," and no spending limit. Three one-line guardrails would have prevented the entire incident. We go deep on the pattern in guardrails for business AI, but the short version is that constraint is what makes autonomy survivable.
Rollout: Shadow Mode, Then Supervised, Then Trusted
Trust in an organization is built in phases, and skipping phases is how agents get uninstalled. Phase one is shadow mode: the agent processes real work but its output goes nowhere, and humans compare its decisions against their own for a couple of weeks. This costs nothing politically and tells you where the eval set was lying to you.
Phase two is supervised: the agent acts, but a human approves each action, ideally in a queue designed to take ten seconds per item rather than ten minutes. Phase three is scoped autonomy: the agent acts alone inside its dollar threshold and its tool permissions, with monitoring watching the edges. Each phase earns the next. The organizations that rush to phase three to impress someone always end up back at phase one, except now with a story people tell at conferences.
The Monitoring Nobody Builds
Agents don't fail like normal software. They fail quietly: a slow drift in output quality as the world changes, a loop that politely emails the same customer nine times, a systematic wrongness on one category of input that nobody notices because the dashboard only shows volume. Production AI agents for business operations need monitoring that hunts for wrongness, because the uptime dashboard will be glowing green the whole time the agent is quietly ruining things.
The minimum viable version is almost embarrassingly human: a standing 30-minute weekly review where someone reads a random sample of the agent's outputs and scores them. Do that and you'll catch drift weeks before your customers do. Skip it and you'll find out from a screenshot on social media. The tooling layer is covered in monitoring AI agents, but the ritual matters more than the tool.
Fifteen of a hundred demos make it to production. The difference isn't model quality, budget, or luck. It's that someone owned evals, guardrails, rollout, and monitoring as real work. That someone is usually an FDE. Now it can be you.