Mar 4
Loading video player...
(31:51-32:55)Building trustworthy, high-quality agents remains one of the hardest problems in AI today. Even as coding assistants automate parts of the development workflow, evaluating, observing, and improving agent quality is still manual, subjective, and time-consuming. Teams often spend hours “vibe checking” agents, labeling outputs, and debugging failures. In this session, Corey Zumar, Staff Software Engineer at Databricks, demonstrates how to use MLflow to automate and accelerate agent observability. Learn how to apply proven patterns to deliver agents that behave reliably in real-world conditions. Key Takeaways and Learnings: 🔹 Understand the development lifecycle of Agent development for better observability 🔹 Use MLflow key components along the development lifecycle to enhance general observability: tracking and debugging, evaluation with MLflow judges, and a prompt registry for versioning 🔹 Select appropriately from a suite of over 60+ built-in and custom MLflow judges for evaluation, and use Judge Builder for automatic evaluation. 🔹 Use MLflow UI to compare and comprehend evaluation scores and metrics 🗓️ Date: February 19, 2026 0:00 MLflow History and Mission 2:00 Challenges in Building AI Agents 4:01 Streamlined Process for Building High-Quality Agents 5:23 Fundamental Components of ML Platform for Agents 10:36 Gathering Feedback with MLflow Labeling 16:39 Discovering Quality Issues with MLflow Assist and Judges 21:36-24:15 Creating and Running Routing Accuracy Judge 24:21 Analyzing Evaluation Report and Creating Datasets 27:23 Fixing Issues with MLflow Prompt Registry and Optimization 31:51 Verifying Fixes and Comparing Agent Performance 34:19 AI Gateway for Governance and Cost Control 34:54 Future Roadmap and Getting Started