RAG Systems

Design and build Retrieval-Augmented Generation systems to query custom data

📋 Table of Contents

• Chapter 1: Introduction to RAG
• Chapter 2: Document Ingestion & Chunking
• Chapter 3: Vector Databases & Retrieval
• Chapter 4: Generation, Advanced RAG & Evaluation

💻 Workshop Practice Notebook

Master all the concepts from this guide with hands-on practice:

• Practice in VS Code: Open the notebook in your local editor. Requires a local .env file containing your API keys.
• Practice in Google Colab: Open the notebook directly in Colab. Setup cells are included to install packages and request API keys.

💻 VS Code | 🚀 Colab | 📥 Download Notebook

Chapter 1: Introduction to RAG

Large Language Models (LLMs) are incredibly powerful, but they have two core limitations: they suffer from hallucinations (fabricating facts convincingly) and their knowledge is static (limited to their pre-training training cut-off date). Retrieval-Augmented Generation (RAG) solves these problems by grounding model answers in private, external data.

1. What is RAG & Why is it Needed?

Instead of relying solely on the LLM’s internal weights to generate answers, RAG queries an external datasource to retrieve relevant documents matching the user’s question, and then passes those documents to the LLM as context.

Why not just fine-tune?

• Real-time Updates: RAG can access live data (like database records or APIs) instantly. Fine-tuning is static and slow.
• Cost & Time: Fine-tuning requires renting GPUs and running training runs. RAG connects to databases dynamically at runtime.
• Access Control: RAG lets you filter documents based on user roles (e.g. Employee A cannot query Employee B’s salary documents). Fine-tuned weights expose all information to all users.
• Verifiability: RAG outputs can cite source documents (e.g., “According to Page 12 of the HR Manual…”), whereas fine-tuned outputs cannot be traced.

2. The 3 Pillars of RAG

Every RAG application follows a standard three-step workflow pipeline:

User Query ──> [ 1. Ingestion (Index Documents) ] ──> [ 2. Retrieval (Find Matches) ] ──> [ 3. Generation (LLM Answer) ]

1. Ingestion: Reading raw documents (PDFs, Markdown, wikis), breaking them into smaller chunks, converting those chunks into vector embeddings, and indexing them in a database.
2. Retrieval: When a user asks a question, the system converts the query into a vector and searches the database to find the top-K most similar document chunks.
3. Generation: The retrieved document chunks are formatted into a prompt along with the user’s question, and sent to the LLM to generate a factual, grounded response.

3. Lexical Search vs. Semantic Search

RAG systems rely on Semantic Search (dense retrieval) rather than traditional keyword-matching search:

• Traditional (Lexical) Search: Looks for exact word matches (e.g. TF-IDF or BM25). If you search for “automobile repair”, it will miss documents containing “car mechanics” because the exact characters do not match.
• Semantic Search: Converts text into vector embeddings representing meaning. It knows that “automobile” and "car" are conceptually close, returning relevant matches even without direct keyword intersections.

4. Practice Exercises

Practice 1: Hallucination Mitigation

Explain how RAG reduces the occurrence of model hallucinations compared to open-ended generation.

Solution

• Open-ended Generation: The model relies on predicting the next token based purely on its training parameters. If it doesn’t know a fact, it continues predicting the most statistically probable next words, generating incorrect facts (hallucinations).
• RAG: The model is restricted to a prompt template instruction (e.g., “Answer the query ONLY using the provided context.”). Since the facts are supplied directly in the prompt, the model acts as a summarization/synthesizer, drastically reducing fabricated claims.

Back to Top

Chapter 2: Document Ingestion & Chunking

The first phase of the RAG pipeline is Ingestion, which handles loading files into memory and partitioning them into semantically cohesive chunks that models can query.

1. Document Loading

Before chunking, raw files (PDFs, Markdown, HTML, JSON) must be loaded into LangChain Document objects containing page_content (text) and metadata (source, page number, title).

LangChain provides specialized Document Loaders for different file types:

• TextLoader: Reads plain text files.
• PyPDFLoader: Parses and reads PDF files page-by-page.
• UnstructuredHTMLLoader: Loads and extracts text clean of HTML tags.
• JSONLoader: Selectively extracts fields from structured JSON files using JSONPath.

2. Chunking Strategies

We cannot feed massive documents to LLMs due to context window limits. Splitting them into topic-focused chunks makes similarity searches far more accurate.

Different chunking strategies are suited for different tasks:

2.1 Character Splitting

Splits text by a fixed character count (e.g., every 500 characters).

• Pros: Simple to calculate.
• Cons: Cuts words, sentences, or paragraphs in half, destroying context.

2.2 Recursive Character Splitting (Recommended)

Splits text using a list of separator characters hierarchically (paragraphs \n\n, then lines \n, then spaces , and finally empty strings "").

• Pros: Keeps paragraphs and sentences intact wherever possible.
• Cons: Still requires tuning chunk sizes and overlap margins.

2.3 Token-Based Splitting

Splits text by token counts instead of characters.

• Pros: Directly matches the LLM’s token limits, preventing context window overflow.
• Cons: Harder to read visually for humans.

2.4 Semantic Chunking

Analyzes the semantic meaning of sentences (using embeddings) and splits text only when there is a significant shift in meaning.

• Pros: Highly accurate; groups topics dynamically.
• Cons: Requires executing an embedding model for every sentence during ingestion, making it slow and computationally expensive.

3. Ingestion Python Implementation

Below is a complete implementation that loads a local text file and chunks it recursively.

3.1 Creating a Sample File

First, let’s write a small sample database policy file (knowledge.txt):

with open("knowledge.txt", "w") as f:
    f.write("""
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Passwords must be at least 12 characters long and changed every 90 days.

Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Lunch hours run from 12:00 PM to 1:30 PM. Clean up your dishes after use.
""")

3.2 Loading and Chunking Code

Now, load this file and split it using RecursiveCharacterTextSplitter:

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load the document
loader = TextLoader("knowledge.txt")
documents = loader.load()

# 2. Configure the Splitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=120,      # Max characters per chunk
    chunk_overlap=20     # Overlap boundary
)

# 3. Perform Ingestion Split
chunks = splitter.split_documents(documents)

# 4. Inspect Results
print(f"Loaded {len(documents)} document.")
print(f"Created {len(chunks)} chunks.")

for idx, chunk in enumerate(chunks):
    print(f"\n--- Chunk {idx + 1} ---")
    print(chunk.page_content)
    print(f"Metadata: {chunk.metadata}")

Output:

Loaded 1 document.
Created 3 chunks.

--- Chunk 1 ---
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 2 ---
Passwords must be at least 12 characters long and changed every 90 days.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 3 ---
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'knowledge.txt'}

4. Practice Exercises

Practice 1: PDF Document Loading Setup

Assume you have a PDF file named "report.pdf". Write the code to load it page-by-page using LangChain’s PyPDFLoader and print the content of the first page.

Instructions:

1. Import PyPDFLoader from langchain_community.document_loaders.
2. Initialize it with "report.pdf".
3. Call .load() to get the document list.
4. Access the first page document and print its page_content.

Solution

from langchain_community.document_loaders import PyPDFLoader

# Initialize the PDF loader
loader = PyPDFLoader("report.pdf")

# Load all pages
pages = loader.load()

# Print first page content
print(pages[0].page_content)
print(pages[0].metadata) # Displays page index number

Back to Top

Chapter 3: Vector Databases & Retrieval

After documents are chunked, they must be converted into numerical vectors using an Embedding Model and stored in a Vector Database for similarity retrieval. By the end of this page, you will have a working semantic search engine project.

1. Vector Spaces & Distance Metrics

An embedding model maps text chunks to coordinate vectors in a high-dimensional space. To retrieve the best matches, the vector database calculates distance metrics between the user’s query vector ($q$) and the stored document vectors ($d$):

• Cosine Similarity: Measures the cosine of the angle between two vectors. It ranges from -1 to 1 (where 1 means identical direction). Ideal for text retrieval because it is independent of document length.
• L2 Distance (Euclidean): Measures the straight-line distance between two points. Closer to 0 means higher similarity.
• Dot Product: Multiplies corresponding coordinates. If vectors are normalized, dot product equals cosine similarity.

2. Ingestion + Vector Search Working Project

Let’s build a working database search project that loads a text document, chunks it, generates embeddings using Gemini, stores them in ChromaDB, and runs semantic query searches.

2.1 Install Dependencies

Run in your terminal:

uv add chromadb langchain-community langchain-google-genai

Ensure your API key is in your environment:

export GOOGLE_API_KEY="your_actual_api_key_here"

2.2 Complete Code Implementation

Save and run this code:

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenAIEmbeddings
from langchain_community.vectorstores import Chroma

# 1. Load API keys
load_dotenv()

# 2. Write and Load Document Data
with open("office_policy.txt", "w") as f:
    f.write("""
Reimbursements: Submit all travel receipts by the 25th of each month.
Office Rules: Do not write passwords on post-it notes. Lock your screen when leaving your desk.
Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate.
""")

loader = TextLoader("office_policy.txt")
documents = loader.load()

# 3. Chunk the documents
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=15)
chunks = splitter.split_documents(documents)

# 4. Generate Embeddings
# text-embedding-004 maps chunks to 768-dimension vectors
embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004")

# 5. Load into ChromaDB (In-Memory Vector DB)
print("Generating embeddings and indexing into ChromaDB...")
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings
)

# 6. Perform a Semantic Similarity Search
query = "What happens if I am sick for three days?"
print(f"\nQuery: '{query}'")

# Retrieve the top semantic match (k=1)
results = vector_db.similarity_search(query, k=1)

# 7. Print Output
print("\n--- Semantic Retrieval Match ---")
for doc in results:
    print(f"Content: {doc.page_content}")
    print(f"Source: {doc.metadata['source']}")

Output:

Generating embeddings and indexing into ChromaDB...

Query: 'What happens if I am sick for three days?'

--- Semantic Retrieval Match ---
Content: Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate.
Source: office_policy.txt

3. Practice Exercises

Practice 1: Search Scope (Top-K)

Modify the search query step in the working project to retrieve the top 2 matches (k=2). Run a query searching for "travel refunds and screen security" and print both returned chunks.

Instructions:

1. Call .similarity_search(query, k=2) on the vector_db object.
2. Iterate through the returned list and print each chunk’s content.

Solution

query = "travel refunds and screen security"
results = vector_db.similarity_search(query, k=2)

for i, doc in enumerate(results):
    print(f"Match {i+1}: {doc.page_content}")

Back to Top

Chapter 4: Generation, Advanced RAG & Evaluation

The final stage of the RAG pipeline is Generation, where the retrieved chunks are formatted into a prompt for the LLM. In production, simple RAG systems must be optimized using Advanced RAG architectures and evaluated using RAG metrics to ensure quality.

1. End-to-End RAG Synthesis Pipeline

We compose the retriever, augmented prompt template, and Chat Model using LangChain Expression Language (LCEL):

import os
from dotenv import load_dotenv
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain.chat_models import init_chat_model

load_dotenv()

# 1. Ingest Data
policies = "Sick leave of more than 2 days requires a medical certificate. Submit travel expense receipts by the 25th."
splitter = RecursiveCharacterTextSplitter(chunk_size=80, chunk_overlap=10)
docs = splitter.create_documents([policies])
embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004")
vector_db = Chroma.from_documents(docs, embeddings)
retriever = vector_db.as_retriever(search_kwargs={"k": 1})

# 2. Define Prompt Template
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using ONLY the provided context:\n\n{context}"),
    ("human", "{question}")
])

llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai")

def format_docs(docs):
    return "\n\n".join([doc.page_content for doc in docs])

# 3. Build RAG Chain
rag_chain = (
    {
        "context": retriever | format_docs,
        "question": lambda x: x
    }
    | prompt
    | llm
    | StrOutputParser()
)

# 4. Invoke
print(rag_chain.invoke("When should I hand in travel receipts?"))

2. Types of RAG Architectures

As RAG applications scale, they transition through three architectural paradigms:

2.1 Naive RAG

The standard pipeline: Ingest $\rightarrow$ Embed $\rightarrow$ Retrieve $\rightarrow$ Generate.

• Limitations: Low retrieval precision (retrieving noise), poor recall (missing details), and model hallucinations if the context is too long.

2.2 Advanced RAG

Introduces optimizations before and after retrieval to improve answer relevance:

• Hybrid Search: Combines keyword search (BM25) with vector search (semantic) to find both exact term matches (e.g. product IDs) and conceptual synonyms.
• Reranking: Uses a secondary Cross-Encoder model to calculate exact relevance scores for the top-N retrieved documents, sorting the most important context to the top before sending it to the LLM.
• Metadata Filtering: Restricts searches to specific tags (e.g. {"department": "HR"} or {"year": 2026}), preventing the retriever from pulling irrelevant documents.

2.3 Agentic RAG

Uses LLMs as agents that decide when to query databases, reformulate queries, and self-correct answers if the retrieved data is insufficient.

3. RAG Evaluation Metrics

To measure RAG performance (using frameworks like Ragas or TruLens), systems are evaluated across two halves of the pipeline:

               ┌───────────────────────┐
               │      User Query       │
               └──────────┬────────────┘
                          │
            Context       │       Answer
           Precision      │      Relevance
         & Context Recall │    & Faithfulness
                          ▼
               ┌───────────────────────┐
               │    Vector Context     │
               └───────────────────────┘

3.1 Retrieval Metrics

• Context Recall: Measures if the retriever found all the necessary facts needed to answer the question.
• Context Precision: Measures if the retrieved chunks are highly relevant, or if they contain too much irrelevant noise.

3.2 Generation Metrics

• Faithfulness (Groundedness): Measures if the LLM’s response is based only on the retrieved context. A high score means no hallucinations.
• Answer Relevance: Measures if the generated response directly answers the user’s question, rather than talking about unrelated topics.

4. Practice Exercises

Practice 1: Identifying RAG Failure Modes

Identify which evaluation metric is failing in the following scenarios:

1. The LLM answers a query by making up facts that were not in the retrieved documents.
2. The user asks about sick leave policies, but the database retriever returns documents about office lunch hours, leading to a blank answer.

Solution

1. Faithfulness (Groundedness) is failing, because the model is hallucinating facts outside of the provided context.
2. Context Recall is failing (and consequently Context Precision is low), because the retriever failed to find the correct sick leave documents.