AI Agents 8 - Evaluation, Cost and Scalability

Description

In this lecture, Dr. Mohammad Ghassemi explains how to evaluate, optimize, and scale AI agents built with large language models (LLMs). Starting from first principles, he outlines when LLMs should be used, how to select and test models, and how to manage cost-performance tradeoffs. Using the problem of extracting scientific knowledge from 64 million papers since 1996, he demonstrates step-by-step strategies to reduce costs from millions of dollars and centuries of compute to minutes and a few thousand dollars—through parallelization, smaller models, and targeted data retrieval. Topics include: * Benchmarking LLMs using leaderboards and custom tests * Practical evaluation methods (human, LLM, and metric-based) * Cost modeling and scalability in real systems * Data and tool management via Model Context Protocol (MCP)