Dataset Creation to Evaluate RAG | LLM as a Judge Explained

Description

Evaluating RAG Workflow with RAGAS, LangGraph & Qdrant | OPIK Tracing & Streamlit Demo In this video, I’ll walk you through how to evaluate your RAG (Retrieval-Augmented Generation) workflow using RAGAS, built with LangGraph and Qdrant, and how to observe and trace it using OPIK before deploying on Streamlit. What you’ll learn: ➡️ Key lessons from building an end-to-end RAG evaluation pipeline ➡️ How to create an evaluation dataset using knowledge graph transforms using RAGAS ➡️ Different ways to evaluate a RAG workflow, and how LLM-as-a-Judge works ➡️ Why binary evaluations can be more effective than score-based evaluations ➡️ RAG-Triad setup for LLM-as-a-Judge, inspired by Jason Liu’s “There Are Only 6 RAG Evals” ➡️ Live demo: Evaluate and monitor your LangGraph RAG workflow built using Qdrant, OpenAI LLM, and Streamlit UI, with OPIK traces TL;DR: Learn how to evaluate, trace, and monitor your RAG pipeline: from dataset creation to OPIK-powered observability — using LangGraph, Qdrant, and Streamlit. 🔗 Source Code: https://github.com/lucifertrj/evals Useful resources: 🔗 Eval Driven Development- EDD: Eval process: https://eugeneyan.com/writing/eval-process/ 🔗 LLM as a Judge by Hamel.dev: https://hamel.dev/blog/posts/llm-judge/ 📞 1:1 Meeting Schedule here: https://topmate.io/tarun_r_jain 🗣️ Join Discord Community: https://discord.com/invite/hEMqtDXCHA Reach out to me: ------------------------ Twitter: https://twitter.com/TRJ_0751 LinkedIn: https://linkedin.com/in/jaintarun75 GitHub: https://github.com/lucifertrj Instagram: https://instagram.com/jaintarun.ai ------------------------ Timestamp: 00:00 Introduction 01:12 Task-1: Find a domain expert 02:10 Task-2: How to create the dataset 04:26 Task-3: Evaluate your RAG using LLM-as-a-Judge and Metrics-based 08:40 Trace and Monitor using OPIK 11:34 Code: Index your data to Qdrant 16:29 Code: Streamlit UI for RAG LangGraph app 21:35 Code: Create your dataset 28:35 Code: Evaluate your RAG workflow using RAGAS @qdrant @LangChain @cometml8614 @streamlitofficial