Large language models are excellent pattern learners but unreliable librarians. Left on their own they synthesize—sometimes brilliantly, sometimes inventively—without a built‑in notion of where facts come from or whether a claim has support in the sources that matter to your organization. Retrieval fixes this. It connects a model’s generative ability to your actual knowledge: policies, manuals, contracts, emails, tickets, code, and data. Done well, retrieval turns a general model into a dependable enterprise assistant that knows when to cite, when to abstain, and how to reflect updates as they happen.
This chapter explains the retrieval stack through the lens of the enterprise knowledge loop—a cycle in which content is ingested, cleaned, chunked, embedded, indexed, retrieved, grounded in generation, and then improved through feedback that flows back into content and indices. We cover embeddings and vector databases; why and when to use retrieval‑augmented generation versus fine‑tuning; how to ingest documents and metadata; how caching and freshness shape cost, latency, and hallucination risk; and how to design security and access control that actually holds under scale and audit.
9.1 Embeddings and Vector Databases
Embeddings: geometry for meaning.
Embeddings map items—sentences, paragraphs, images, tables, code snippets—into numeric vectors such that semantic closeness corresponds to geometric closeness. Two maintenance manuals describing the same procedure should land near each other even if they use different words; likewise for a policy and a user’s question about it. Good embeddings compress messy language into a space where nearest‑neighbor search becomes a proxy for relevance.
Choosing an embedding model.
Selection is not cosmetic; it determines recall, multilingual behavior, and whether your distance metric makes sense.
Domain fit. A general web‑trained model may miss legal, clinical, financial, or manufacturing nuance. If domain coverage is crucial, prefer a model adapted to similar corpora or plan a light domain fine‑tune on your material.
Instruction following. Some embedding models accept a task hint (“passage retrieval,” “title retrieval,” “code search”). These often outperform generic embeddings because they are trained to align queries with passages, not just passage with passage.
Multilingual or cross‑lingual. If your documents and queries span languages, choose a model trained to place different languages in the same vector space so a French query can find an English policy (and vice versa).
Modality. For images, charts, and diagrams, use a model that aligns text and vision; the resulting joint space enables “find images that match this description” and “answer questions about this figure.”
Dimension and precision. Higher‑dimensional vectors can capture more nuance but cost memory and bandwidth; lower‑dimensional vectors speed serving. Quantization (e.g., 8‑bit) reduces footprint with minimal quality loss for many workloads; test empirically.
Distance metrics and training alignment.
Cosine similarity (angle), dot product (angle + magnitude), and Euclidean distance (straight‑line) are the usual measures. Use the metric the embedding model was trained to optimize; mismatched choices quietly degrade recall.
Vector databases: indexes for high‑dimensional search.
A vector database stores embeddings and metadata and answers nearest‑neighbor queries fast enough for interactive systems. Architecturally, it provides:
Indexes. Approximate nearest‑neighbor (ANN) structures (graph‑based, inverted files with product quantization, or hybrids) deliver sub‑second search over millions to billions of vectors. Each index family trades memory and build time for query speed and recall; choose based on dataset size, update rate, and latency targets.
Metadata filtering. Retrieval must respect access control, language, date, region, document type, version, and other facets. Efficient pre‑ and post‑filters prune the candidate set before expensive vector comparisons.
CRUD at scale. Enterprise retrieval is not static. The store must handle frequent upserts and deletes, rebuild or incrementally maintain indexes, and support versioned collections for safe rollouts.
Multi‑tenancy and isolation. Either isolate tenants into separate collections/clusters or enforce strong row‑level security with per‑tenant filters and encryption. Treat indices as sensitive databases, not caches.
Observability. Measure recall proxies, tail latencies, memory usage, index health, and filter selectivity. Retrieval failures are often silent; instrumentation is your early warning.
Hybrid retrieval beats monoculture.
Lexical search (e.g., BM25) excels when queries contain specific terms (SKU, clause number, error code). Vector search excels when language varies. Most production systems use a hybrid: retrieve with both, merge or re‑rank. Sparse learned retrievers (which produce “lexical” vectors) can bridge the gap by making keyword‑like features trainable.
Bi‑encoders vs. cross‑encoders.
Bi‑encoders embed queries and passages independently, enabling fast ANN search. Cross‑encoders take a query and a candidate passage together to compute a precise relevance score, but are too slow to apply to the whole corpus. A standard pattern is two‑stage retrieval: use a bi‑encoder to pull the top N candidates, then re‑rank those N with a cross‑encoder before handing the best K to the generator.
Common failure modes and mitigations.
Hubness and anisotropy. In high dimensions, a few vectors can appear spuriously “near” many items. Normalize embeddings, consider dimension reduction, and monitor for hubs (passages that show up everywhere).
Synonym gaps and abbreviations. Domain vocabulary (“GL”, “goods‑lift”, “general ledger”) confuses general models. Add a glossary to the ingestion pipeline and include synonym expansions or use query rewriting to normalize.
Short queries. One‑word queries are ambiguous. Expand queries with clarifying context (“You asked about ‘charge’. Do you mean billing, electrical, or legal?”) or apply a learned query rewriter to add likely terms.
Missing modality. Many relevant answers live in tables, figures, or code blocks. Extract structure (CSV for tables, AST for code) and embed appropriately; attach the original for citation.
9.2 Retrieval‑Augmented Generation (RAG) vs. Fine‑Tuning
Two ways to adapt models to your world.
You can teach a model by changing its weights (fine‑tuning), or you can remind it by providing relevant context at inference time (RAG). The right choice depends on what is changing and how often.
RAG for facts and freshness.
Use RAG when the content you care about—policies, prices, procedures, design docs—changes frequently or must be cited. RAG keeps models small and inference cheap while reflecting updates as soon as the index refreshes. It also provides a natural path to abstention: “I did not find a source for that.”
Fine‑tuning for form and behavior.
Fine‑tune to control style, format, tool use, and domain reasoning patterns (e.g., how to extract a clause, how to summarize a call, how to structure a ticket in your schema). Fine‑tuning is durable; it captures how to answer, not what the answer is. Parameter‑efficient techniques (adapters, low‑rank updates) reduce cost and simplify rollback.
When to do both.
Most enterprise assistants combine the two: fine‑tune for task competence and schema adherence; retrieve for facts and citations. This division of labor yields better reliability than trying to “bake in” dynamic knowledge via repeated fine‑tunes.
Trade‑offs summarized in prose.
Latency and cost. Fine‑tuned models can be faster because they need less context; RAG introduces retrieval latency but reduces hallucinations and lets you use smaller models. With careful engineering (caching, tight chunks, hybrid retrievers), RAG can meet interactive budgets.
Governance. RAG is easier to audit. You can point to sources and show access controls in action. Fine‑tuning requires dataset documentation and re‑evaluation per update.
Failure profiles. Fine‑tuned models fail silently when content changes; RAG fails with “no good retrieval,” which you can detect and act upon.
Data needs. Fine‑tuning requires labeled examples of “good answers.” RAG requires clean, deduplicated, well‑chunked content with metadata. Pick the easier investment for your organization first; add the other as you mature.
A decision heuristic.
If stakeholders ask, “Where did this come from?” or “Is this the latest policy?”, start with RAG. If they ask, “Why can’t the model stick to our format and tone?”, add fine‑tuning. If they ask both, do both.
9.3 Document Ingestion, Chunking, and Metadata
Retrieval depends more on how you prepare content than which LLM you use. Ingestion is an engineering discipline: connectors, parsers, normalizers, chunkers, and taggers organized into a repeatable, observable pipeline.
Connectors and extraction.
Pull content from source systems—document repositories, wikis, ticketing tools, code hosts, email archives, data warehouses—using APIs or change‑data capture (CDC). Preserve source IDs and versioning. For PDFs and scans, use high‑quality OCR; for slides and spreadsheets, extract reading order and cell structure; for websites, strip navigation chrome and capture canonical URLs. If you cannot rebuild everything nightly, choose connectors that stream deltas.
Normalization and cleaning.
Standardize encodings, collapse duplicated whitespace, and remove spammy boilerplate. Keep headings and section boundaries; many chunkers rely on this structure. For emails and tickets, strip signatures and legal footers; for code, remove binaries and vendor directories; for wiki pages, resolve transclusions and templates to yield the content a reader sees.
De‑duplication.
Duplicates inflate index size, slow search, and bias retrieval toward repeated content. Apply exact hashing to eliminate byte‑identical copies; use shingling or embedding similarity to remove near‑duplicates (e.g., same policy with different headers). Retain one canonical copy per logical document per version.
Chunking: the forgotten art.
The chunk is your retrieval atom. Too large and you return irrelevant text; too small and you lose context. Strategies:
Length‑based with overlap. Split at N tokens with 10–20% overlap to maintain coherence across boundaries. Simple and robust.
Structure‑aware. Split along headings, bullet lists, code blocks, or table rows. Preserve semantic units (a procedure step, a clause, a function) rather than arbitrary spans.
Semantic or discourse‑based. Use a segmenter to detect topic shifts and split at boundaries. Effective for long reports, manuals, and wikis.
Hierarchical. Create small chunks and also maintain parent chunks (section, chapter). At query time, retrieve at one level and “zoom” as needed.
Decontextualization. Prepend each chunk with a synthesized title or path (“From Policy 3.2.1—Expense Reimbursements—Meals”) so it stands alone when quoted. This raises comprehension and reduces hallucinations.
Metadata: retrieval’s scaffolding.
Attach rich metadata to each chunk. At minimum: title, section path, author/owner, source system and ID, version, effective and expiry dates, language, document type, jurisdiction, data classification, and permissions. For code: repo, branch, commit, license. For tables: column names, units, and primary keys. Metadata enable filtering (“only policies effective today”), ranking (“prefer latest version”), and security.
Indexing strategy.
Use collection versioning: build a new index for each batch, validate, then atomically switch traffic. This avoids half‑built states. For streaming updates, maintain a write‑optimized staging index and periodically merge into the read‑optimized main index. Keep embedders stable within an index; mixing models reduces comparability. If you must upgrade the embedding model, rebuild; avoid incremental mixing unless you also store the embedding model ID with each vector and query with the matching model.
Quality gates.
Before a new collection goes live, run checks: duplication rate, orphan chunks (no source), chunk length distribution, metadata completeness, and a small retrieval battery (“Given query X, do we get Y?”). Observability here saves days chasing “model regressions” that are actually content bugs.
9.4 Caching, Freshness, and Hallucination Reduction
Retrieval systems are living systems. Performance comes from caching, fidelity from freshness, and trust from hallucination controls that prefer silence to speculation.
Caching layers.
Embedding cache. Cache embeddings of queries and documents to avoid re‑encoding. If you use instruction‑tuned embedders, include the instruction string in the cache key; small differences change vectors.
Retrieval result cache. Cache the top‑K results for frequent queries (after applying the same filters). Normalize queries (case, punctuation, stop‑words) to increase hit rate. Set conservative TTLs if content changes often; attach the collection version to the key so cache invalidates on switch.
Generation cache. Cache full responses for read‑mostly workflows (FAQ, policy Q&A). Tie cache entries to specific retrieval contexts (document IDs + snippets) and model/prompt versions. Beware of cache poisoning: never store responses to untrusted prompts without review in public‑facing systems.
Feature caches. Cache re‑ranking scores and entailment checks; these are deterministic functions of query–passage pairs and are expensive to recompute.
Freshness mechanics.
Ingestion cadence. Match update frequency to content volatility: real‑time for tickets and incidents; hourly/daily for wikis and policies; weekly for manuals. If you can, stream deltas via CDC and micro‑batch indexing.
Invalidation. On document update or permission change, re‑chunk and re‑embed; mark old chunks as tombstoned. Invalidate caches keyed by document ID. For soft changes (metadata only), you may skip re‑embedding.
Temporal ranking. Prefer the latest effective version in ranking; demote content past expiry. Keep prior versions to answer “what was the rule last quarter?” when audits require.
Freshness signals. Expose document timestamps to the generator (“Context includes v3.2.4 updated May 2025”) so it can phrase with time awareness (“as of May 2025”).
Hallucination controls.
Answerability detection. Before generating, decide whether the query is answerable with retrieved content. If not, ask a clarifying question or respond with “no source found” rather than invent.
Strict grounding. In instruct prompts, require the model to answer only from the provided passages and to cite them. Penalize unsupported sentences during preference tuning; reward explicit abstention.
Schema and validation. For extraction and data entry, require JSON schemas and validate fields. Reject and retry with a stricter prompt when violations occur; escalate to a human rather than post‑hoc “fix.”
Quotation and attribution. Encourage short quotes with citations for high‑stakes statements. For policies and contracts, quoting reduces paraphrase drift and increases user trust.
Compression and synthesis. When multiple passages conflict, instruct the model to report the conflict and cite both, rather than averaging them into a false certainty.
Tool usage. For numeric or database questions, route to calculators and SQL rather than hoping the model “remembers” a figure. Logged tools add verifiability.
Cost–latency balance.
Caches cut cost and latency dramatically; conversely, aggressive freshness can blow budgets if you rebuild constantly. Categorize content by volatility and importance; set different cadences and TTLs by category. Track tokens per dollar end‑to‑end; retrieval that reduces prompt length by 30% pays for itself quickly at scale.
9.5 Security and Access Control in RAG Systems
A retrieval system is a security boundary. It surfaces internal knowledge to an LLM and to users. Design for “least privilege” from the first line of code.
Entitlements flow to chunks.
Permissions start at the source system (DMS, wiki, code host) and must propagate through ingestion to each chunk and vector. Store ACLs or ABAC attributes with every chunk. At query time, compute the user’s effective permissions and filter candidates before similarity scoring wherever possible (pre‑filtering scales better and reduces side‑channel leakage).
Isolation choices.
Hard isolation. Separate indices per tenant or sensitivity tier. Simple mental model; higher hardware cost; smoother audits.
Soft isolation. Single index with per‑row filters and encryption. Lower cost; demands rigorous testing, query plans that push down filters, and explicit proofs that no cross‑tenant results can leak via caches or debug endpoints.
In both cases, segregate caches by tenant and sensitivity. The most common breach class in RAG prototypes is cross‑tenant response caching.
Authentication and authorization.
Gate every layer—API gateway, orchestrator, retriever, vector database—with service‑to‑service authentication and short‑lived tokens. Pass the user context (subject, roles, attributes) end‑to‑end. Avoid “superuser” service accounts for routine retrieval; use them only for backfills and administration.
Prompt injection defenses.
Treat retrieved content as untrusted. Adversaries can hide instructions inside documents (or links) to make the model ignore your system prompt or exfiltrate data. Controls:
Content sanitization. Strip or neutralize HTML/markdown that can hide instructions; block external link following unless explicitly allowed.
Instruction separation. Use structured prompts with clear fields for “system policy,” “user question,” and “retrieved content.” Instruct the model to ignore instructions inside retrieved content; reinforce this in SFT.
Tool allow‑lists. If the model can call tools, restrict accessible functions per request based on the user and context. Never allow arbitrary shell/HTTP without explicit policy.
Context size discipline. Long, loosely filtered contexts increase attack surface. Retrieve narrowly and re‑rank rigorously; prefer a few strong passages over a heap.
PII and sensitive data.
Scan incoming content for personal and sensitive categories; tag and minimize. Apply stricter retrieval policies for sensitive chunks (e.g., only to specific roles; no external sharing; no logging of raw content). In generation, run output filters to detect PII leakage before response. For regulated data, consider on‑prem or VPC‑hosted inference to avoid cross‑border exposure.
Encryption and key management.
Encrypt vectors and metadata at rest; keep encryption keys in a managed KMS with rotation and least‑privilege policies. Use TLS for all links. For high sensitivity, consider field‑level encryption of metadata (at the cost of filter performance).
Audit and forensics.
Log: the authenticated user, query text (appropriately redacted), filter attributes, retrieved chunk IDs and sources, model and prompt versions, citations in the response, tool calls, and final output. Retain logs per policy; secure them against tampering. Auditors will ask not only “who saw what?” but “how did this answer get produced?” Your logs should answer both.
Supply‑chain and vendor risk.
If you use third‑party embedding APIs or hosted vector databases, review data handling terms: training rights, retention, subprocessor lists, breach notification, data residency, and deletion guarantees. For open‑source components, track versions and CVEs; vector stores and ANN libraries are performance‑critical and must be patched promptly.
Multi‑region and sovereignty.
Route retrieval and inference to regions that match user and data jurisdiction. Keep indices regional; avoid cross‑region context assembly unless policy allows. For sovereign sectors, deploy fully within accredited facilities and keep management planes isolated.
Human-in-the-loop with access control.
Escalated reviews should inherit the requester’s permissions: a reviewer can only see what the original user was entitled to see. Build review tools that fetch context under a delegated token, not under a superuser. This prevents “review mode” from becoming an exfiltration vector.
