The project died on a Tuesday, in a one-line email from the compliance officer: "Client data does not leave this building. Full stop." Nine months of discovery, a working demo, an ops team that had already named the thing, all of it pinned to the wall by eleven words and a period with anger issues. If you found this post by typing "self host LLM for private data" into a search bar right after reading an email like that, you're in the right place.

Then it un-died. The fix wasn't a cloud vendor with a longer PDF of certifications. It was an open-weight model running on a GPU box in a closet that also stored, for reasons nobody could explain, a broken office chair and four thousand branded pens.

So here's the short version, before the honest caveats: you should self host an LLM for private data when regulation, contract terms, or a threat model you can defend out loud say the data can't leave your control. Open-weight models on modest hardware genuinely deliver on narrow tasks like extraction, classification, and summarization. What they won't do is match a frontier API at everything, and pretending otherwise is how closet servers become expensive space heaters.

When the API is not an option

Three triggers actually justify the self-hosted route, and only one of them is the one vendors pitch you.

The first is regulation, the real kind with statute numbers. Health data, certain financial records, anything where "we sent it to an API in another jurisdiction" is a sentence you say to a regulator with your lawyer present. The second is contracts: plenty of client agreements name approved subprocessors, and "a model API you've heard of" is often not on the list. The third is what I call paranoia with a point: trade secrets, bid pricing, the recipe. If a leak would end the business rather than embarrass it, the paranoia is just risk management wearing a trench coat.

Notice what's not on the list: a vague feeling that cloud is sketchy. Zero-retention enterprise API tiers exist, and for a lot of businesses they're genuinely sufficient. The test is whether a specific document (a law, a signed agreement, a board policy) says no. If the objection can't be quoted, it can usually be negotiated.

What modest hardware actually gets you

Say you're a 60-person firm. The realistic build is one workstation-class box: a GPU with 24 to 48 GB of memory, decent CPU, lots of RAM, an illustrative $6,000 to $10,000 all-in. On that box you run quantized open-weight models in the 7 to 14 billion parameter range, and they answer in seconds.

Here's what that actually covers, from projects I've watched survive contact with production:

And here's what it doesn't cover: multi-step agentic reasoning, subtle code generation, and long-context synthesis at frontier quality. That gap has a price tag, and the next section names it. If you're deploying this on the client's own metal, the playbook for running Docker on client hardware applies verbatim: containerize the model server, pin the versions, and make the upgrade path boring.

The honest capability gap

Time for illustrative numbers, clearly labeled as illustrative because every model release shuffles the deck. On tightly-scoped extraction tasks with validation, a well-prompted local 8B model might land around 90% pass rate where a frontier API lands 96%. On short summarization, the gap narrows further. On "read these 40 pages, find the contradiction, and explain the business implication," the gap is a canyon, and no amount of prompt embroidery bridges it.

The counterintuitive fix is that the local model usually wins on engineering, not raw smarts. A local 8B with a strict JSON schema, a validator, and one retry will beat a frontier model with a vibes-based prompt in production, because production is where vibes go to die. You narrow the task, decompose the hard parts, and verify the outputs mechanically. The model does the fuzzy 80%; deterministic code does the judgement-y 20%.

There's also a middle rung worth knowing: a rented GPU running a 70B-class open model, which closes much of the reasoning gap without buying hardware. It still counts as self-hosted for many policies because the weights and logs stay in an environment you control. Ask the compliance officer before assuming, obviously. Their emails are memorable.

The architecture that keeps you sane

One decision matters more than which model you pick: put the model behind an internal HTTP API, OpenAI-compatible, from day one. Your app calls http://llm.internal/v1/chat and has no idea what's behind it.

Swap-ability is the feature

Start with the 8B that fits your box. When the workload outgrows it, swap in a 70B on rented GPU by changing a config line. When a task turns out to be non-sensitive after all, burst that one to a hosted API. Because the boundary is an API, every upgrade is an ops task instead of a rewrite. The rest of the stack stays ordinary: a retrieval layer over your documents, a queue for long jobs, and logs that live in your building. This slots into the same FDE stack essentials you'd use anywhere, with the model endpoint swapped for a closet. The public-facing web layer, if you have one, can still live on the usual platforms; the Vercel vs Cloudflare tradeoffs don't change just because the brain is on-premise.

Two boring requirements save you later: version your prompts and eval sets alongside the model, and wire up the same monitoring you'd give any small app, because a model that silently degrades is worse than one that loudly crashes.

Cost and maintenance reality

The CFO version, again illustrative: $7,000 of hardware amortized over three years is about $200 a month. Power and cooling for a hard-working GPU, call it $40 to $80. Someone's time for patching, driver updates, model upgrades, and the occasional 3 a.m. fan noise investigation, realistically two to four hours a month. All-in, you're often under $500 a month for steady-state operation.

Compare that with an API bill that scales with usage. Low volume, the API wins on cost and it's not close. High, steady, predictable volume, the closet box pulls ahead, sometimes within a year. The crossover isn't the point, though; if you're self-hosting, cost is usually the third reason, after "the policy said so" and "we'd rather not think about it."

The real cost is that it's a server, not a pet rock. Someone owns it. If that sentence has no name attached at your company, that's the actual decision you need to make.

When to self host LLM for private data: the checklist

Run through these in order, and stop at the first yes:

  1. A quotable document forbids external processing. Law, signed contract, or board policy, in writing. Self-host.
  2. A leak would be existential, not embarrassing. The formula, the bid prices, the client list. Self-host, or at least self-host those flows.
  3. Volume is high and steady, and tasks are narrow. Extraction, classification, summarization. Self-hosting pays for itself.
  4. None of the above. Use a zero-retention enterprise API tier and spend your engineering budget on the actual product.

The honest summary of how to self host LLM for private data: less often than the hardware vendors claim, more often than the API vendors admit, and almost always narrower than the first grand plan. Pick the two or three flows that genuinely need the closet server, build them behind an API you can swap, and let everything else be someone else's uptime problem. Your compliance officer will frame the email. Mine did.