Chunking, embedding models, metadata filtering | Learn LLMs in Sinhala Ep. 23

Description

Retrieval-Augmented Generation (RAG) is transforming the way Large Language Models (LLMs) interact with knowledge. In this video, we break down three critical pillars of building an effective RAG pipeline: Chunking, Embedding Models, and Metadata Filtering. What you’ll learn in this session: - Introduction to RAG – Why it matters, how it solves hallucination and knowledge cutoff issues. - Chunking Strategies – Fixed-length, sentence-based, paragraph-based, sliding window, semantic, and recursive approaches. Learn which strategy to pick for your use case. - Embedding Models – Key factors (context window, dimensionality, tokenization, cost, efficiency) and a review of popular models like OpenAI, BGE, E5, Jina, and SFR-Embedding. - Metadata Filtering – How structured attributes (date, author, topic, etc.) improve retrieval accuracy, efficiency, and relevance.