Jan 16
Loading video player...
What’s next for AI observability? Marc Klingen (Langfuse) and Jason Lopatecki (Arize AI) dig into the future of tracing, evals, and monitoring in a panel led by Mastra CEO Sam Bhagwat. They unpack what observability really means for LLMs and AI agents: why tracing comes before evals, how online and offline evaluations differ, and where each shines. The panel tackles OpenTelemetry as a potential standard, the gap between AI-native observability and traditional monitoring, and the messy realities of dev vs. production. They also dive into longer-running agents, multi-agent systems, and what it all means for the next generation of AI. Marc Klingen is Co-Founder and CEO of Langfuse, an open-source LLM engineering platform backed by Y Combinator (W23). Langfuse offers traces, evals, prompt management, and metrics to debug and improve LLM applications. The platform has over 19,000 GitHub stars and is used by 19 of the Fortune 50 and 63 of the Fortune 500 companies. Jason Lopatecki is Co-Founder and CEO of Arize AI, a platform for AI and agent observability and evaluation. He is a garage-to-IPO executive with extensive experience in scaling teams and building machine learning and data processing architectures. Arize AI provides development tools, evaluation frameworks, and observability solutions for production-ready AI applications and agents. 🔗 CONNECT WITH MARC KLINGEN LinkedIn: https://www.linkedin.com/in/marcklingen Twitter/X: https://x.com/marcklingen Langfuse: https://langfuse.com 🔗 CONNECT WITH JASON LOPATECKI LinkedIn: https://www.linkedin.com/in/jason-lopatecki-9509941 Arize AI: https://arize.com 🔗 CONNECT WITH SAM BHAGWAT (MODERATOR ) LinkedIn: https://www.linkedin.com/in/calcsam Twitter/X: https://x.com/calcsam Mastra: https://mastra.ai 📚 MASTRA RESOURCES Mastra: https://mastra.ai Learn Mastra in the world's first MCP-Based Course: https://mastra.ai/course Principles of Building AI Agents (Book): https://mastra.ai/book Patterns for Building AI Agents (New Book): https://mastra.ai/blog/patterns-book https://docs.google.com/forms/d/e/1FAIpQLSduJjc515f6RZJqtkR2ByqJZrB0iP8B7SUKnjjZE9IajH_I8w/viewform The TypeScript AI conference is the first conference for TypeScript AI developers. The inaugural event took place on November 6, 2025, in San Francisco and online. MASTRA? Mastra is an open-source TypeScript framework designed for building and shipping AI-powered applications and agents with minimal friction. It supports the full lifecycle of agent development—from prototype to production. You can integrate it with frontend and backend stacks (e.g., React, Next.js, Node) or run agents as standalone services. If you’re a JavaScript or TypeScript developer looking to build an agentic or AI-powered product without starting from first principles, Mastra provides the scaffolding, tools, and integrations to accelerate that process. 00:00 Intro 01:15 Defining Observability: Evals vs. Traces 03:20 Online and Offline Evals 05:50 Audience Poll 06:31 OpenTelemetry (OTEL) for Tracing: Pros and Cons 09:00 AI-Native Observability vs. Traditional Tools 11:37 Observability in Development vs. Production 14:55 Longer-Running Agents and Observability 17:40 Future Trends: Automation and Full Workflow Integration 19:32 Iterating Prompts Without Regression 21:37 Sampling Metrics in Production 23:12 Optimization and Automated Prompt Tuning 25:58 Observability in Multi-Agent Systems