The email arrived at 7:12 on a Monday with the subject line "quick question about the API invoice." Finance had found it, and the number was 4.1x the month before, which had itself been described as "a bit higher than expected."
Nobody had done anything wrong, which is the annoying part. The AI feature launched, users liked it, usage grew, and every one of those happy users was clicking a button that cost somewhere between a fraction of a cent and, occasionally, forty cents. Success, billed monthly.
Teams that reduce LLM API costs in production don't do it with a single trick, and they definitely don't do it by turning the feature off. They do it with four boring levers applied in order, and the usual result is a 40 to 70 percent cut that users never notice (illustrative range, but a common one).
Here's the short version: route every request to the cheapest model that can actually handle it, cache anything you've answered before, put your prompts on a diet, and set hard budgets with alerts that page a human. Do those four things and the surprise invoice becomes a flat, boring line item.
Why Token Bills Sneak Up on You
Token bills have four favorite hiding spots. Retry loops, where a flaky call quietly runs three times. Context creep, where every new feature adds just one more document to the prompt until you're paying for a novella per request. Agent loops, where the model calls the model calls the model, and each lap costs money. And my personal favorite: dev traffic, where your own team's testing gets billed at production rates because nobody set up a separate key.
The first step is embarrassingly simple: log every call with its cost. One wrapper function, one database table, one week of data. You cannot diet what you do not weigh, and most teams have never once looked at cost per feature. The graph always produces one audible gasp.
A worked example from that first week of logging, with illustrative numbers that match the usual shape: one document-summary feature turned out to be 80 percent of spend, and inside it, the retry policy accounted for a third of its calls. A flaky upstream timeout meant the whole prompt, all 6,000 tokens of it, ran twice or three times on every unlucky request. The fix was a longer timeout plus a retry with jitter, and it beat every clever prompt tweak we tried that month.
Reduce LLM API Costs in Production With a Model Ladder
Most apps send every request to the flagship model because that's what the tutorial used. This is like hiring a surgeon to apply band-aids. Set up a ladder instead: a small cheap model for classification and extraction, a mid-tier model for drafts and summaries, and the flagship only for requests that fail a quality check or match a known-hard pattern. Illustrative prices that shift monthly but keep the shape: something like $0.15, $1.50, and $15 per million output tokens across the tiers.
Model routing rules can be dumber than you think. "If the input is under 500 tokens and the task is extracting five fields, use the small model" covers an absurd share of real traffic. In one ops-copilot build, 70 percent of requests never needed to leave the bottom rung; users noticed nothing except faster answers. The same pattern powers the AI copilot for operations teams, where most questions are genuinely easy and only look impressive.
Keep a small eval set so you can prove the cheap model is good enough before and after the switch. Evals for LLM features are how you defend the downgrade to your own anxious brain at 2 AM.
Build the escape hatch before you need it: when the cheap model's confidence is low or the output fails a schema check, retry one rung up, not straight to the flagship. Most ladders leak because every uncertainty rockets to the most expensive tier. Log how often each rung promotes traffic upward; if the bottom rung bails out more than 30 percent of the time, your routing rule is wrong, not your model.
Caching the Boring Questions
Users are creatures of habit. An FAQ-facing assistant gets the same twenty questions wearing different hats, roughly 60 percent repeats in one deployment I measured (illustrative, but typical). An LLM caching strategy starts with an exact-match cache on the normalized prompt: it costs nothing and answers instantly. A semantic cache, which stores embeddings and reuses answers for near-duplicates, catches most of the rest.
Two rules keep the cache honest. Include the model name and prompt version in the cache key, or you'll serve stale answers after an upgrade and confuse everyone. And never cache anything personalized or permission-scoped; "what's my invoice balance" is not a cache candidate, it's a compliance incident with extra steps.
Set a TTL that matches how fast the underlying truth changes. Product documentation answers can live for a week. Anything touching live operational data gets minutes, not days, or gets skipped entirely. When in doubt, cache the expensive retrieval step instead of the final answer; fresh words over stale data beats stale words every time.
Prompt Diets and Context Budgets
Open your longest system prompt and count the tokens. If it's over 2,000, you're paying a cover charge on every request for instructions the model skim-reads anyway. Prompts accrete: legal adds a disclaimer, product adds tone guidance, someone pastes the company values doc in its entirety. Audit quarterly and cut like you're editing for a hostile magazine.
Retrieval context needs the same treatment. Stuffing ten chunks into the prompt when three would do triples your input cost for a marginal quality gain. Measure everything: tokens in, tokens out, cost per request, per feature. Once you see that one feature spends 80 percent of its budget on context it barely uses, the diet writes itself.
The most profitable audit I ever ran took one afternoon. A support assistant's system prompt had grown to 3,400 tokens, including a full page of tone rules the model ignored and a product changelog nobody remembered adding. Cutting it to 900 tokens dropped cost per request by a fifth and, per the eval set, slightly improved the answers. Shorter instructions get followed; longer ones get skimmed, same as humans.
Budgets, Alerts, and Kill Switches
Set a monthly budget per feature, not per company. When the document Q&A feature crosses its number, it degrades gracefully: smaller model, shorter context, or a polite "high demand, try again shortly." What it must never do is keep spending silently until finance sends another 7:12 AM email.
Token budget alerts at 50, 80, and 95 percent of the monthly cap catch runaway loops while they're still cheap. A runaway agent loop can burn a week of budget in an afternoon; the scars from that story are documented in what it takes to run AI agents in production, including the part where the fix is a counter and an if statement.
The Week One Cost Audit
If your bill just spiked, do this in order. One: wrap every call with cost logging. Two: rank features by spend. Three: move the top feature's easy traffic down the model ladder. Four: add an exact-match cache to the chattiest endpoint. Five: set budgets with alerts. That's one week of work, and it typically pays for itself before the next invoice arrives.
The goal was never the smallest possible bill. It's a boring bill: flat, predictable, and small enough that the feature's value gets measured in outcomes instead of apologies. Boring is beautiful. Boring ships.