Mar 16
Loading video player...
Learning Objective Learn how to implement comprehensive observability for generative AI workloads on Amazon Bedrock and Bedrock AgentCore. Understand the evolution from simple chatbots to agentic AI systems and master the unique challenges of monitoring non-deterministic AI agents in production environments. What We Covered - Evolution of AI Systems: The progression from basic chatbots to generative agents and agentic AI systems with increasing autonomy and business impact - Observability Challenges: Unique challenges for agentic AI including non-deterministic control flows, complex root cause analysis, and holistic system health assessment - Trust Framework: Key evaluation questions for AI agents - can they do the job, achieve goals, make right decisions, use best tools, and provide accurate answers - Amazon Bedrock AgentCore Evaluations: Fully managed service for continuous AI agent quality assessment with built-in evaluators, custom templates, and CloudWatch integration - Monitoring & Dashboards: Comprehensive metrics including sessions, traces, error rates, latencies, and end-to-end tracing capabilities for full visibility from prompt to response - Continuous Improvement Loop: Process for enhancing AI agents through test case definition, execution, evaluation, gap analysis, and iterative improvements - Production Deployment Path: Best practices for moving from offline evaluation to shadow mode, AB testing, and full production rollout 📺 CloudOps Enablement YouTube Playlist - https://www.youtube.com/playlist?list=PLx0P9lMoFCUYwTMOyCi6IGjB9DxMXygZi 🗓️ Enablement Series Cloud Operations Enablement Series - https://aws-experience.com/amer/smb/events/series/Cloud-Operations-Enablement 🛠️ GitHub Repositories - AWS OTEL Python Instrumentation - https://github.com/aws-observability/aws-otel-python-instrumentation/ - One Observability Demo (Waggle AI) - https://github.com/aws-samples/one-observability-demo/ - Sample Amazon CloudWatch GenAI Observability - https://github.com/aws-samples/sample-amazon-cloudwatch-generative-ai-observability - AgentCore Observability Tutorial - https://github.com/awslabs/amazon-bedrock-agentcore-samples/tree/main/01-tutorials/06-AgentCore-observability - AgentCore Evaluations Tutorial - https://github.com/awslabs/amazon-bedrock-agentcore-samples/tree/main/01-tutorials/07-AgentCore-evaluations 🧪Workshops - CloudOps GenAI Observability Workshop - https://catalog.workshops.aws/cloudops-genai-ob11y/en-US - AWS One Observability Workshop - https://catalog.workshops.aws/observability/en-US 📚 Documentation - Observability Best Practices – Useful Links https://aws-observability.github.io/observability-best-practices/resources/useful-links/ - Bedrock AgentCore Observability Docs - https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html - CloudWatch GenAI Observability Docs - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/GenAI-observability.html - CloudWatch Pricing - https://aws.amazon.com/cloudwatch/pricing/