The support bot looked fine on paper. Response times under two seconds. Uptime at 99.9%. Every API call returning a cheerful 200 OK. But somewhere around Tuesday afternoon, it started confidently telling customers their orders had shipped when they absolutely had not. Nobody noticed until Thursday, when a frustrated customer emailed a screenshot directly to the CEO. Three days of quiet wrongness. No alerts fired. No dashboards turned red. The agent was healthy, but its answers were garbage.

This is the problem with trying to monitor AI agents in production the same way you monitor a web app. Traditional observability tells you whether the server is breathing. It does not tell you whether the model has drifted, whether a prompt injection has turned your helpful assistant into a coupon-code vending machine, or whether a context-window overflow is causing the agent to forget its own instructions halfway through a task. You need a different telemetry stack for agents, one that watches behavior, not merely infrastructure.

To monitor AI agents in production effectively, track four signals that traditional application monitoring misses entirely: output quality drift, cost per interaction, error patterns that are not HTTP errors, and latency spikes that precede timeouts. Infrastructure uptime tells you the service is running; these four signals tell you whether the agent is still worth trusting. If you only watch the server, you will find out about problems from your customers. And they will not be gentle about it.

The Silent Failure Problem

Software usually fails loudly. A database connection drops and throws an exception. An API returns a 500. You see the smoke, you find the fire. AI agents fail like a bad employee on a remote team: they keep showing up, keep responding, and slowly get worse at their job without anyone realizing.

A typical mid-size firm we worked with deployed a customer-service agent that routed questions and drafted replies. For the first two weeks, it was magical. Then the product team launched a new pricing tier and updated the knowledge base. The agent kept citing old prices. Not because the API was down, but because the retrieval pipeline was pulling from a stale vector store. The error rate in the traditional sense was zero. The accuracy rate had fallen off a cliff.

The lesson: agents need behavioral monitoring, not merely systems monitoring. You are not asking "is it online?" You are asking "is it still right?" Those are different questions with different answers.

What to Monitor

After a few painful incidents, we narrowed the signal list to four metrics that actually predict agent trouble.

Latency per turn. At one client, the average turn took 800ms until a product manager added three detailed examples to the system prompt. It jumped to three seconds overnight. Not total request time, but the time from user input to first token, plus the time between reasoning steps if your agent chains multiple calls. A spike here usually means the context window is bloated, the model is doing extra reasoning it should not be doing, or a retrieval step is hanging. If your typical turn takes 800ms and it suddenly hits three seconds, something changed in the prompt or the context.

Cost per interaction. LLM billing is token-based, which means costs can creep in ways that CPU-based services do not. A prompt that grew by 400 tokens because someone added a long system message does not change the API response code, but it changes your burn rate. We track cost per session and alert if the weekly average moves more than 15% without a known cause. It catches prompt bloat, retry loops, and runaway context windows before they become budget problems.

Error rate, redefined. For agents, an error is not a 500. It is a failed tool call, a structured-output parse failure, a refusal that should not have happened, or a hallucinated parameter sent to a downstream API. We log these as "agent errors" and track them separately from infrastructure metrics. A 2% agent error rate is usually worth investigating.

Output quality. This is the hardest and the most important. We run a weekly sample of 30-50 conversations through a lightweight eval: did the agent answer the question? Did it cite the right source? Did it stay within policy? You cannot automate all of this, but you can spot-check enough to catch drift. One team we worked with caught a policy violation in 12% of sampled chats. That is not an edge case. That is a pattern.

Building a Lightweight Dashboard

You do not need a $50k observability suite to watch an agent. A scheduled Python script that logs to Postgres and emails a daily summary covers most small-team needs. The key is putting the four signals in one place where someone looks at them.

Our default setup: a single dashboard with four tiles. Tile one shows average latency per turn over the last 24 hours with a threshold line. Tile two shows cost per interaction, trending weekly. Tile three shows agent error rate, split by error type. Tile four shows the result of the most recent quality sample pass rate. Red means below threshold. Yellow means watch it. Green means move on.

The dashboard lives on a URL the team checks every morning. Not because it is beautiful, but because it is honest. When the quality sample tile turned yellow for three days straight at one client, we caught a retrieval issue before it became a customer-facing disaster. That tile alone has paid for every minute spent building the dashboard.

If you want to get fancier, you can pipe structured logs into a tool like cleaning messy data before it reaches your agent, but start simple. A cron job, a Postgres table, and a basic web page beat an elaborate pipeline that never ships.

Alert Fatigue vs. Real Problems

One fast way to ruin monitoring is to alert on everything. If your phone buzzes because one user got a slow response at 3am, you will stop looking at alerts entirely. We use a simple rule: alert on trends, not points.

Latency spikes for five minutes? Log it, do not page anyone. Cost per interaction up 20% for three consecutive days? That is worth a Slack message. Quality sample pass rate below 85% for two weeks? That is a meeting. The goal is to tune alerts so you wake up for the right reasons, not every reason.

We also separate "agent health" alerts from "business health" alerts. Agent health is latency, error rate, cost. Business health is conversion rate, customer satisfaction, ticket volume. When both move in the same direction, you have a real problem. When only agent health moves, you have a technical issue. When only business health moves, you might have a product issue. Keeping them separate prevents the panic that comes from conflating a slow model with a broken business.

The Cost Creep Nobody Talks About

Token spend has a way of growing 20% week-over-week without anyone noticing. It is never one big change. It is a system message that got longer, a retrieval chunk that started returning more context, a new instruction example that added 300 tokens to every prompt. Each change is tiny. The compound effect is not.

At one client, the monthly LLM bill doubled in six weeks. The team was convinced the provider had raised prices. What happened: a well-meaning product manager had added three detailed examples to the system prompt to "improve consistency." Each example was 400 tokens. Multiplied by 10,000 daily interactions, that was $800 a week in new spend. The fix took ten minutes. Finding it took two hours of cost-per-interaction tracing.

We now review prompt token counts in the same way we review code changes. Any system prompt change over 100 tokens gets a cost estimate in the pull request. It sounds bureaucratic until it saves you $40,000 a year.

When to Escalate to a Human

No agent should operate without an escape hatch. The question is when to pull it. We use three rules for graceful degradation.

First, if the agent error rate exceeds 5% for more than an hour, fallback to a human queue. Second, if a user explicitly asks for a human twice in the same session, bypass the agent entirely. Third, if the agent detects high emotional sentiment, offer a human handoff proactively. These rules are not perfect, but they keep users from feeling trapped.

The best agent deployments we have seen treat escalation as a feature, not a failure. A customer who gets a fast, honest handoff remembers the brand positively. A customer who fights with a broken bot for twenty minutes remembers it as a betrayal. The cost of a human handling one conversation is trivial compared to the cost of a churned customer telling their friends about the terrible AI.

If you are shipping agents to production, the telemetry you build is as important as the model you choose. Start with the four signals, add a lightweight dashboard, tune your alerts for signal over noise, and always give your users a way out. The agents that survive in production are not the smartest ones. They are the ones that fail gracefully, get caught early, and improve continuously. That does not happen by accident. It happens because someone decided to ship AI agents to production with eyes wide open.

And if you are wondering whether your data is clean enough to support any of this, start with the basics. A messy pipeline will sabotage even the best monitoring setup, which is why we always chat-with-your-data interface patterns after we have fixed what is underneath them.