How to Improve Quality of Multi-Agent Systems with Agent Bricks

Description

In this video, you will learn how to build, evaluate, and deploy multi-agent systems on Databricks using only natural language. This video demonstrates creating a "Novel Ideas Bookworm" multi-agent system, which combines an AI Genie (for structured data/inventory), a RAG chatbot (for book recommendations/knowledge), and an MCP server (for general web search) to provide intelligent bookstore assistance. See how to continuously improve the multi-agent system's output quality by using natural language feedback from labeling sessions and built-in evaluation metrics. 🎯 What You'll Learn: Key Agent Use Cases: Information Extraction, Knowledge Assistant (RAG chatbot), AI/ML Genie for structured data, Multi-Agent Supervisor (RAG + Genie), Custom LLM, and fully custom agents. Build Multi-Agent Systems: Use Databricks to build and improve quality on multi-agent systems using SME feedback and natural language. Evaluate with Traces and Custom Evals: Use the "Experiment" tab for detailed interaction traces and the "Evaluations" tab for built-in metrics (e.g., correctness, relevance) and custom evaluation prompts. ⏱️ CHAPTERS: 00:00 Agent Bricks Overview – Build, evaluate, and deploy multi-agent systems on Databricks with natural language. 02:53 Architecture: Vibe-Coding Stack – Overview of the Novel Ideas multi-agent system: 02:53 AI Genie (Inventory Space) – Uses real-time inventory and sales data for bookstore assistance. 04:00 Knowledge Assistant (Bookstock Bot) – RAG chatbot for personalized book recommendations. 05:50 Multi-Agent Supervisor (Bookworm) – Orchestrates Bookstock Bot, Inventory Space, and Tavly MCP for broader queries. 08:51 Improving Quality – Use “Improve Quality” to label sessions, add expectations, and merge expert feedback. 12:20 Wrap-Up & Next Steps 12:47 Using the MCP Server – Routes general questions to Tavly MCP when not in references. 13:42 Evaluating Performance – Use “Experiment” for traces, “Scores” for custom evals, and “Evaluations” for metrics like correctness and relevance. 15:55 Conclusion – Build and evaluate a multi-agent system in minutes with natural language on Databricks. 📚 Resources: • GitHub repo - https://github.com/databricks • Documentation: Agent Bricks - https://www.databricks.com/product/artificial-intelligence/agent-bricks AWS - https://docs.databricks.com/aws/en/generative-ai/agent-bricks Azure - https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-bricks/ 🤝Stay connected: LI Connect with us: Website: https://databricks.com Twitter: https://x.com/databricks/ LinkedIn: https://www.linkedin.com/company/databricks/ Instagram: https://www.facebook.com/databricksinc/ Facebook: https://www.instagram.com/databricksinc/