How to Prepare Enterprise Documents for RAG

Preparing enterprise documents for RAG means turning messy files into clean, self-contained, well-labeled chunks: extract accurate text (OCR for scans), chunk along the document’s own structure, keep tables and headings intact, and attach metadata to every chunk. Most disappointing RAG deployments fail here, not in the model. If the right passage was never extracted cleanly, never chunked sensibly, or carries no metadata to filter on, no retriever and no language model can recover it. This guide covers the ingestion pipeline that decides whether retrieval-augmented generation works on an institution’s real archive.

Why does document preparation decide RAG accuracy?

Document preparation decides RAG accuracy because retrieval operates on chunks, not on documents. The model only ever sees the handful of passages the retriever returned, so the quality ceiling is set at ingestion time. A chunk that splits a policy clause in half, loses its heading, or scrambles a table cannot support a correct, citable answer no matter how strong the embedding model or the LLM is.

The working test for any chunk: it should stand alone in front of a reader who has never seen the source file. A chunk that reads “the limit is 5 lakh per transaction” with no indication of which product, which policy version, or which section it came from will retrieve for the wrong questions and produce a citation nobody can verify. Document preparation is the work of making each chunk independently meaningful and independently attributable.

How do you handle scanned PDFs and OCR quality?

Scanned PDFs are images, so a RAG pipeline that skips OCR indexes nothing from them. Optical character recognition creates the text layer that chunking and embedding depend on, and its error rate propagates into every downstream stage.

Practical rules for OCR in an enterprise ingestion pipeline:

Detect, do not assume. Check whether each PDF already has an extractable text layer. Re-OCRing a born-digital PDF often produces worse text than extracting it directly.
Scan quality matters more than the OCR engine. Low-resolution, skewed, or noisy scans degrade every engine. Deskew, denoise, and prefer 300 DPI source images where you control scanning.
Match the language and script. Documents in Hindi, Tamil, or other non-Latin scripts need OCR models trained for them. Mixed-language pages, such as English forms with regional-language annotations, need explicit handling.
Guard the identifiers. OCR errors in account numbers, policy IDs, statute citations, and clause references are the most damaging kind, because exact-token matching depends on them. A “0” read as “O” quietly makes a document unfindable, which is why the choice between vector and keyword search in enterprise RAG interacts with OCR quality.
Keep a confidence signal. Where the engine reports per-page or per-token confidence, store it. Low-confidence pages are the first place to look when retrieval mysteriously misses.

Handwriting, stamps, signatures, and multi-column forms remain the hardest cases. For regulated archives, flag those pages for human review rather than indexing unreliable text that the assistant will later cite with confidence.

What chunk size and overlap should you use for RAG?

Chunk size controls one tradeoff: smaller chunks retrieve more precisely, larger chunks carry more context. A small chunk produces a focused embedding that matches a specific question well but may omit the sentence that gives the answer meaning. A large chunk carries surrounding context, but its embedding averages several topics together, so it matches many questions weakly instead of one question strongly.

A reasonable starting point for enterprise prose is roughly 200 to 500 tokens per chunk with 10 to 20 percent overlap, tuned afterwards from measurement rather than intuition. Overlap exists to stop an answer being severed at a boundary. It costs index size and introduces near-duplicate results, so keep it modest.

Three signals say the chunk size is wrong:

Retrieved chunks are on-topic but the answer is cut off mid-explanation. Chunks are too small, or overlap is too little.
Retrieval returns long passages where one sentence is relevant and answers drift. Chunks are too large.
Different document types clearly need different settings. That is normal, not a failure. A dense contract and a slide deck should not share one configuration.

Which chunking strategy should you choose for enterprise RAG?

Chunking strategy matters more than chunk size. This table compares the approaches used in enterprise RAG pipelines.

Strategy	How it splits	Best for	Main failure mode
Fixed-size (character or token)	Every N tokens, ignoring structure	Uniform plain text, quick baselines	Cuts sentences, clauses, and tables in half
Recursive character	Paragraph, then line, then word, then character, until size fits	General-purpose default for mixed prose	Still structure-blind at document level
Structure-aware (heading-based)	Along headings, sections, clause numbers	Policies, manuals, contracts, SOPs	Needs reliable heading extraction; sections can be huge
Layout or element-aware	By parsed element: title, paragraph, table, list, figure	Scanned and complex PDFs, forms, reports	Depends on parser quality; costlier to run
Semantic	At points where embedding similarity drops	Narrative text with no headings	Compute cost; boundaries can be unstable
Parent-child (small-to-big)	Retrieve small chunks, pass the larger parent to the model	Precision plus context, long reference documents	More index and plumbing complexity

For most regulated document sets (circulars, policies, clinical protocols, academic regulations, contracts), structure-aware chunking with a parent-child retrieval pattern is the strongest default: match on the tight child chunk, answer from the full parent section, cite the parent. Fall back to recursive character splitting only where structure genuinely does not exist.

How do you preserve tables and layout in RAG chunks?

Tables are where naive ingestion fails most visibly, because flattening a table to plain text destroys the row-and-column relationship that carries the meaning. A rate card, a fee schedule, or a lab reference range becomes an ambiguous stream of numbers.

Preserve tables with four rules:

Keep the table whole. Where size permits, one table equals one chunk. Splitting a table across chunks is what produces answers pairing the right number with the wrong row.
Repeat the header. If a long table must be split, carry the header row and any unit row into every fragment.
Serialize to markdown or HTML, not to space-aligned plain text, so cell relationships survive into the model’s context.
Add a caption. Prefix each table chunk with its title, its section, and its units (percent, INR lakh, per annum).

The same logic applies to layout generally: headings, list nesting, footnotes, and page numbers are meaning, not decoration. Preserving the heading path (for example, “Retail Lending Policy > Section 4 > 4.2 Overdraft charges”) and prepending it to each chunk is one of the cheapest accuracy improvements available in a RAG pipeline.

What metadata should every RAG chunk carry?

Metadata turns a chunk from a floating passage into a governable, filterable, citable unit. Attach it at ingestion, because reconstructing it later is expensive.

At minimum, every chunk should carry:

Source identity: document title, file identifier, page or section number, and a stable link back to the original file.
Heading path: the chain of headings above the chunk, prepended into the chunk text and stored as a field.
Version and date: effective date, revision number, and status (current, superseded, draft). This is how you stop the system citing a repealed circular.
Ownership and classification: department, document type, sensitivity level.
Permission tags: the roles, teams, or rows allowed to see this chunk, enforced at retrieval time and not only in the interface.
Language: so multilingual collections can be filtered or routed correctly.

Metadata pays off twice: as a pre-filter that narrows the search space before ranking runs, which raises precision, and as the substrate for access control. Retrieval that ignores permission metadata can surface a passage to someone who should never see it, which in a bank or hospital is a breach, not a bug.

How do you handle duplicates, versions, and document updates?

Enterprise archives are full of near-duplicates: the same policy as a DOCX, a PDF export, an email attachment, and a scanned signed copy. Left unmanaged, duplicates crowd the top-k results, so the model reads one passage four times and never sees the document that answers the question.

Hash at document and chunk level. Collapse exact-duplicate chunks to one indexed entry with multiple source references.
Detect near-duplicates. Use similarity thresholds to flag copies that differ only cosmetically, keep the authoritative one, and mark the rest.
Make version explicit. Prefer the current effective version by default, retain superseded versions only where regulation requires it, and tag them so the assistant can state which version it is quoting.
Re-index incrementally. On change, re-chunk and re-embed only the affected document, and delete its old chunks in the same operation. Orphaned chunks from a previous version are a common source of confidently wrong, correctly cited answers.
Expire, do not abandon. Retired documents must leave the index, not merely disappear from a folder listing.
Re-measure. Any significant ingestion change moves retrieval quality, so re-run your labeled test set as described in how to evaluate RAG accuracy.

What should you check before indexing a corpus?

Run this checklist before an enterprise corpus goes into production retrieval:

Every file type in scope has a tested extraction path (native PDF, scanned PDF, DOCX, XLSX, PPTX, HTML, email).
OCR output is spot-checked on the worst-quality sample, including non-Latin scripts.
Chunking follows document structure, and the heading path is prepended to chunk text.
Tables are chunk-whole, header-repeated, and captioned.
Every chunk carries source, page or section, version, date, classification, permission tags, and language.
Duplicates are collapsed and superseded versions are marked.
An update path exists that deletes old chunks when a document changes.
A labeled question set exists to measure retrieval before and after ingestion changes.

How does Samvad AI ingest an institution’s own documents on-premise?

Samvad AI is a source-cited RAG assistant that answers only from your own documents and runs on-premise, air-gapped, or hybrid by configuration, so the full ingestion pipeline (extraction, OCR, chunking, embedding, indexing) executes inside your infrastructure and no document leaves your network. Samvad AI cites the exact source passage for every answer and says plainly when an answer is not in your sources, which makes ingestion defects visible rather than silent: a chunk that lost its table header or heading path shows up in the citation, where a reviewer can catch it.

Because Samvad AI is permission-aware at the role and row level, multilingual, and backed by a tamper-evident audit log, the metadata you attach at ingestion is enforced at retrieval rather than merely displayed. To plan an ingestion pipeline for your own archive, see Samvad AI for secure document question answering or write to teclops.ai@gmail.com.

Frequently asked questions

What is the best chunk size for RAG?

There is no universal best chunk size, but roughly 200 to 500 tokens with 10 to 20 percent overlap is a sound starting point for dense retrieval over enterprise prose. Tune it against a labeled set of your own questions rather than copying a default, and expect contracts, manuals, and slide decks to need different settings.

Do you need OCR for scanned PDFs in a RAG pipeline?

Yes. A scanned PDF is an image, so without optical character recognition its text layer is empty and retrieval will never find it. Run OCR at adequate resolution, use models trained for the scripts in your archive including Hindi and other non-Latin scripts, and spot-check output, because OCR errors in account numbers or clause references silently break exact-token retrieval.

How do you handle tables in RAG documents?

Keep each table in a single chunk where size allows, repeat the header and unit rows into every fragment if a large table must be split, and serialize to markdown or HTML instead of space-aligned plain text so row and column relationships survive. Prefix each table chunk with its title and units so the model does not misread the numbers.

How do you update a RAG index when documents change?

Track a content hash and version per document, re-chunk and re-embed only the documents whose hash changed, and delete the superseded chunks in the same operation so retired text cannot be retrieved and cited. Mark withdrawn documents as expired rather than leaving them indexed, and re-run retrieval evaluation after each significant re-index.

How do you chunk spreadsheets, slide decks, and email threads for RAG?

Treat each format by its natural unit: a spreadsheet by sheet and table region with headers carried into every chunk, a deck by slide with the slide title and any speaker notes attached, and an email thread by individual message with sender, recipients, and date as metadata. Chunking these formats by raw character count is the most common cause of unusable retrieval on mixed enterprise archives.