Text Analysis Tutorial: RAG Basics (indexing, retrieval, prompt construction; small corpus)

Title & Overview

Template: RAG Basics: An Intermediate, End-to-End Analysis Tutorial
Overview (≤2 sentences): Learners will implement a Retrieval-Augmented Generation (RAG) pipeline with indexing, retrieval, and prompt construction over a small corpus. It is intermediate because it emphasizes reproducibility, retrieval–generation trade-offs, and structured evaluation of retrieval quality and answer correctness.

Purpose

The value-add is teaching learners how to combine retrieval with generation in a defensible way. They will explore indexing strategies, prompt design, and evaluation of RAG systems, while stressing reproducibility and guardrails for small-scale deployment.

Prerequisites

• Skills: Python, Git, pandas, ML basics.
• NLP: embeddings, retrieval, generation models, evaluation metrics.
• Tooling: pandas, scikit-learn, Hugging Face Transformers, Sentence-Transformers, FAISS, Haystack, MLflow, FastAPI.

Setup Instructions

• Environment: Conda/Poetry (Python 3.11), deterministic seeds.

• Install: pandas, scikit-learn, Hugging Face Transformers + Datasets, Sentence-Transformers, FAISS, Haystack, MLflow, FastAPI.

• Datasets:

  • Small: WikiMovies (QA corpus).
  • Medium: HotpotQA (subset for multi-hop retrieval).

• Repo layout:

  tutorials/t11-rag-basics/
    ├─ notebooks/
    ├─ src/
    │   ├─ indexer.py
    │   ├─ retriever.py
    │   ├─ generator.py
    │   ├─ pipeline.py
    │   └─ config.yaml
    ├─ data/README.md
    ├─ reports/
    └─ tests/
  

Core Concepts

• RAG workflow: corpus indexing → document retrieval → prompt construction → generation.
• Indexing choices: exact vs approximate nearest neighbors (FAISS).
• Prompt construction: retrieved docs + query → model input; guardrails to prevent overflow.
• Evaluation: retrieval precision@k, generation accuracy/F1, calibration of confidence.
• Error slicing: by query type, document length, retrieval difficulty.

Step-by-Step Walkthrough

1. Data intake & preprocessing: load WikiMovies/HotpotQA subsets; reproducible splits.
2. Indexing: build FAISS index on sentence/document embeddings (Sentence-Transformers).
3. Retrieval baseline: BM25 vs dense ANN embeddings; compare retrieval quality.
4. Generation baseline: DistilBART/T5-small with retrieved docs as context.
5. Evaluation: retrieval precision@k, MRR; generation EM/F1; slice performance by query type.
6. Error analysis: missed retrievals, hallucinations, overly long prompts.
7. Reporting: metrics tables, retrieval/generation error cases → reports/t11-rag-basics.md.
8. (Optional) Serve: FastAPI endpoint with retrieval + generation pipeline, schema validation, max token guardrails.

Hands-On Exercises

• Ablations: BM25 vs FAISS; top-k=1 vs top-k=5 retrieved docs.
• Robustness: add distractor documents; measure retrieval/generation accuracy.
• Slice analysis: compare single-hop vs multi-hop queries.
• Stretch: hybrid retrieval (BM25 + embeddings reranker).

Common Pitfalls & Troubleshooting

• Retriever failure cascades: poor retrieval → bad generation.
• Long prompts: exceeding model max tokens; must truncate or window.
• Metrics misuse: only reporting generation accuracy; retrieval must be evaluated separately.
• Index drift: embeddings change → FAISS index rebuild required.
• OOM issues: dense embeddings on large corpus; mitigate with batching.

Best Practices

• Log both retrieval and generation configs in MLflow.
• Keep reproducibility: fixed seeds, dataset fingerprints, tokenizer artifacts.
• Unit tests: deterministic retrieval results under fixed embeddings.
• Guardrails in serving: enforce schema, max context length, reject empty queries.
• Baseline → Retriever Comparison → RAG pipeline narrative.

Reflection & Discussion Prompts

• Why does retrieval improve generation robustness?
• How does index choice (BM25 vs FAISS) affect performance and scalability?
• What risks come from hallucinations in RAG pipelines?

Next Steps / Advanced Extensions

• Integrate cross-encoder rerankers for retrieval.
• Multi-hop retrieval pipelines.
• Domain adaptation: civic datasets (public documents, meeting notes).
• Lightweight monitoring: retrieval recall drift over time.

Glossary / Key Terms

RAG, retrieval, generation, ANN, FAISS, BM25, precision@k, MRR, hallucination.

Additional

• [Sentence-Transformers](https://www.sbert.net/)
• [Haystack](https://docs.haystack.deepset.ai/)
• [Hugging Face Transformers](https://huggingface.co/docs/transformers)
• [Hugging Face Datasets](https://huggingface.co/datasets)
• [FAISS](https://faiss.ai/)
• [MLflow](https://mlflow.org/)
• [FastAPI](https://fastapi.tiangolo.com/)

Contributors

Author(s): TBD
Reviewer(s): TBD
Maintainer(s): TBD
Date updated: 2025-09-20
Dataset licenses: WikiMovies (open), HotpotQA (CC BY-SA).

Issues Referenced

Epic: HfLA Text Analysis Tutorials (T0–T14).
This sub-issue: T11: RAG Basics.

Resources