The Slack message lands at 2 a.m., or first thing Monday, which feels the same: "The AI thing is saying weird stuff again." You shipped the document summarizer six weeks ago. It demoed beautifully. Now a customer is pasting screenshots into a support ticket, and you're staring at your own creation like it's a raccoon in the kitchen — you invited it in, technically, but you have no idea what it's been doing in there.

This is the state of LLM monitoring in production for most small teams. Which prompt version produced that output? No idea. Last week's cost is a mystery too, hiding inside an invoice with a bump you're afraid to investigate. The feature works until someone asks how you know.

The good news is that the fix is smaller than the vendors want you to believe. Effective LLM monitoring in production for a small team comes down to three questions: what did it do, what did it cost, and was it any good. You can answer all three with a database table, a thin logging wrapper, and a weekly ritual. No platform purchase required. Yet.

The 2 AM Slack Message

Let's be honest about how most small teams ship LLM features. You prototype against the API, tune the prompt until the demo stops embarrassing you, wrap it in a try/except, and deploy. Observability is `console.log` and optimism. This is not a moral failing; it's what shipping fast looks like. But it leaves you blind in three specific ways, and the 2 a.m. message is just the moment you find out.

Blind spot one: no traces. When the model goes weird, you can't reconstruct the inputs, the retrieved context, or which of your four prompt iterations was live. Blind spot two: no cost attribution. The bill is one number, so the retry loop somebody added in a hurry can 10x your spend and you'll find out from accounting. Blind spot three: no quality signal. Users experience degradation for weeks before anyone complains, because complaining takes effort and quietly working around your feature doesn't.

Enterprise observability suites exist to solve all of this, and they're priced like they know it. But here's the secret: the suites are overkill. The questions aren't.

LLM Monitoring in Production: What a Small Team Actually Needs to See

Strip it down and there are exactly three questions you need to answer on demand:

Everything else, from dashboards to alerting hierarchies to eval frameworks with their own DSLs, is refinement of these three. If you're running AI agents in production, the same three questions apply, just with more steps per trace. Nail the trio and you're ahead of most teams ten times your size, which is a fun sentence to be able to write.

The Shoestring Stack

Here's the setup you can build in a day, and yes, a day is realistic. First: one table (or collection, or even a big append-only log) that records every LLM call — timestamp, feature name, user ID, model, prompt version string, input tokens, output tokens, latency, cost in micro-dollars, and a pointer to the full request and response. That table is your monitoring platform. Everything else is decoration.

Second: a thin wrapper around your LLM client, maybe 40 lines, so every call goes through one function that writes to the table before returning. If you'd rather not write the wrapper, grab one of the open-source tracing libraries — several good ones exist, and the honest criterion is "does it write structured rows I can query without learning their query language." If the answer involves their cloud dashboard, keep walking.

Third: a weekly sampling ritual, calendar invite and all. Thirty minutes, every Friday, review a sample of outputs. More on that below. The stack is deliberately boring — the same philosophy as choosing a boring script over an agent when the logic is deterministic. Boring is what survives your next launch crunch.

Cost Tracking That Catches the Runaway

An illustrative story, assembled from real-shaped incidents. A small team ships a support-drafting feature. Week one: $40. Week two: $45. Week five: $612. The culprit, when someone finally queries the table: a well-meaning retry-on-empty-response loop combined with a system prompt that had quietly grown to 3,000 tokens of "helpful context." Every hiccup retried, every retry re-sent the whole preamble, and the loop's cap was set to, wait for it, twenty.

The fix was twenty minutes. The discovery took three weeks because nobody was looking at cost per feature. With per-feature and per-user token accounting, this story ends in week five on Monday morning: a query shows the support feature's spend up 12x week-over-week, you group by user, find that one account triggered 4,000 retries, and push a fix before lunch.

Budgets and alerts are fifty lines of code: a nightly job sums cost by feature, compares against thresholds you picked during a calm moment, and posts to Slack when something crosses the line. Not sophisticated. Extremely effective. The whole apparatus pays for itself the first time it catches a runaway, and it will catch one.

Quality Without a Labeling Team

Quality monitoring feels like it needs an eval framework and a labeling pipeline. It doesn't, not yet. It needs a habit: sample twenty outputs a week, stratified across your features, and have a human (you, on Friday, with coffee) mark each one good or bad with a one-line note. Track the rolling good-rate per feature in your trusty table. That's it.

What you're watching for is movement, not absolutes. The summarizer holds at 85% good for a month, then dips to 70%. Something changed: the model provider updated, a prompt got edited, the input distribution shifted. Now you investigate with actual traces instead of vibes. When the score moves significantly, that, not a blog post about evals, is your signal to build a proper eval set for that feature. Until then, twenty samples and honest thumbs will catch anything your users would have caught three weeks earlier.

One addition that punches above its weight: an in-product feedback control users will actually touch. Not a survey — a thumbs-up/thumbs-down on the output itself, with an optional one-liner. Response rates will be low and the signal will still be gold, because thumbs-down with a note is a user doing your sampling for free. If you're doing RAG over company data, these notes also double as a list of retrieval failures to fix.

When to Graduate to Real Tooling

The shoestring stack has honest limits, and pretending otherwise is how teams end up with a homegrown observability platform they hate maintaining. Graduate when any of these become true: you're past roughly a million calls a month and ad-hoc queries on one table get slow; you're juggling multiple models and providers and need normalized comparison; compliance starts asking who saw what data and you need retention policies and access controls, not a Postgres row policy you wrote on a Friday.

At that point, evaluate the real platforms on the same three questions: traces, cost attribution, quality signal. Buy the one that answers them with the least lock-in. And keep your table — the discipline you built, logging every call with feature and user attached, transfers directly. The platforms ingest exactly that shape of data. Your shoestring wasn't a prototype of monitoring; it was the training montage.

Start tonight. One table, one wrapper, one Friday calendar invite. The next "weird stuff" message will arrive eventually (it always does), but this time you'll open a query instead of a prayer. If you want somewhere to point the numbers once they exist, a dashboard people actually open is a good second build.