RAG for Company Data: A Field Guide

June 27, 2026 · 6 min read · AI & Data

Key Takeaways

RAG fails on document mess, not model choice.
Spend the first eighty percent of effort on ingestion and dedupe.
Enforce permissions at query time, every time.
Grade fifty real questions weekly or you're flying blind.
Split retrieval misses from generation misses before fixing anything.

Then you pointed it at the real SharePoint. Forty thousand files, half of them scans, a third named some variant of final_FINAL_v7_ACTUAL.docx, and folder permissions last reviewed sometime during the second Bush administration. The bot began hallucinating procurement policy from a 2016 marketing deck. Legal got involved. It went poorly.

Welcome to retrieval augmented generation on company data, where the model is the easy part and the documents are the boss fight.

RAG works when the plumbing is boring and solid: clean ingestion, sane chunking, permissions enforced at query time, and a weekly evaluation habit. It fails when teams obsess over model choice while feeding the system a landfill. If your RAG pilot disappointed you, the fix is almost never a different LLM. It's the ninety percent of the project that happens before the model ever sees a token.

The demo that lied to you

Every vendor demo runs on twenty hand-picked documents with real titles and no duplicates, and every production environment is forty thousand files of entropy. This isn't malice; it's survivorship bias with a sales quota. The demo corpus hides every hard problem you're about to meet: six copies of the same policy with different dates, scanned contracts that are technically PDFs the way a photocopy of a steak is technically dinner, and a folder called 'Archive 2019' that everyone is afraid to delete.

The mental adjustment is simple. The question was never 'can an LLM answer questions about clean documents?' It demonstrably can. The question is whether your documents can be made answerable, and that is an engineering project wearing a chat window as a hat.

Retrieval augmented generation is a plumbing problem

On real projects, the effort split looks roughly like this, and it's illustrative but directionally honest: document cleanup and dedupe take about 45% of the work, chunking and evaluation about 25%, permissions about 20%, and model selection and tuning about 10%. Teams invert this at their peril, spending week one benchmarking models and month two discovering that 'the model is dumb' actually means 'the retrieval found a 2014 draft.'

Model choice is the least interesting decision you'll make. Any frontier model answers well when handed the right chunks; none answer well when handed the wrong ones. Buy the boring plumbing first.

The ingestion pipeline that survives reality

A pipeline that survives your actual document estate does four things. It dedupes, both exactly (same file, four folders) and near-enough (same policy, three 'final' versions; keep the newest, flag the rest). It OCRs the scans, because half your institutional knowledge is trapped in images of paper. It keeps metadata with every chunk: owner, date, and system of record, so answers can say 'per the 2024 policy' and users can sanity-check. And it syncs incrementally, because a corpus that requires a full re-index on every change is a corpus that silently goes stale.

This is where the first eighty percent of effort lives, and it's the same discipline as any first data pipeline built on company data: boring, idempotent, and honest about what it couldn't parse. If your documents need heavier lifting, the patterns in LLM document processing apply before anything gets chunked.

Chunking is taste, not science

Practical defaults: 400 to 800 tokens per chunk with 10 to 15% overlap for prose. Keep tables whole or convert them to something readable; a table sliced in half is misinformation. Split on document structure first (headings, sections) and token counts second. FAQs and policy Q&As go one complete item per chunk, no exceptions. Then test against real questions and adjust, because the 'optimal' chunk size is whatever makes your fifty eval questions pass.

Permissions are the product

Here's the scenario that ends RAG projects: a summer intern asks the bot a casual question and receives a cheerful summary of the executive compensation spreadsheet, which happened to be in the crawl. Nobody gets fired, but the project gets buried so deep it achieves fossil status. Legal has a long memory and a short list of second chances.

The rule is absolute: retrieval must enforce access controls at query time, per user, every time. Map identity from your SSO to document-level ACLs, filter before anything reaches the model, and log what was retrieved for whom. 'We'll clean up permissions after the pilot' means the pilot is the last phase, just not the way you hoped.

Evaluate or die

Collect fifty real questions from staff (not from the engineering team; from the people who will actually ask), each with a known good answer and the document it lives in. Grade the bot weekly. The crucial habit is splitting failures into two buckets: retrieval misses, where the right chunk never surfaced, and generation misses, where it surfaced and the model still answered wrong. Retrieval misses mean fixing data, metadata, or chunking. Generation misses mean fixing prompts or models. Teams that don't split the buckets fix the wrong thing with great confidence.

This weekly grading loop is the same operating discipline as running AI agents in production: vibes are not a metric, and a score moving from 70% to 90% over a month is what progress actually looks like.

What a first month looks like

Week one: pick one department and one corpus, and freeze the scope in writing. Week two: build the ingestion pipeline, dedupe, OCR, and metadata. Week three: wire permissions to your SSO and run the first eval pass; expect to be humbled, that's the point. Week four: pilot with ten friendly users, a feedback button, and a standing weekly review of misses.

By day thirty you'll have something no demo can show you: a system that answers questions about your documents, with your permissions, improving on a schedule. The model was ready in 2023. The plumbing is ready when you build it. Start with the folder everyone is afraid to delete.

Frequently Asked Questions

What is RAG in plain terms?

Search, then summarize. The system retrieves the most relevant chunks of your documents and hands them to the language model, which writes an answer grounded in those chunks instead of answering from memory. The model is the voice; your index is the brain.

Why does our RAG bot make things up?

Usually retrieval failure. The right chunk wasn't found because of duplicates, bad chunking, or missing documents, so the model improvises with confidence. Check what was retrieved before blaming what was generated.

How much does a company RAG system cost to run?

For a mid-size corpus, embedding and LLM usage typically runs $50 to $500 per month. The real cost is engineering time on ingestion, permissions, and evaluation, which is where the project actually lives.

Should we fine-tune instead of using RAG?

Almost never for knowledge Q&A. Fine-tuning teaches style and behavior, not facts, and your facts change weekly. RAG keeps knowledge in the index, where you can update it without retraining anything.

Want the full picture? Read the complete guide to forward deployment engineers.

Read the Guide