Apr 9
Loading video player...
Join this L300 session to discover how to build comprehensive observability across your entire GenAI stack: from GPU utilization and infrastructure health to agent decision flows, tool invocations, and application performance. CloudWatch OTLP for Metrics - Leverage OpenTelemetry Protocol (OTLP) for standardized metrics collection - Configure CloudWatch to receive OTLP metrics from distributed GenAI workloads - Enable vendor-agnostic observability across your infrastructure Infrastructure Observability - Monitor inference workloads on Amazon EKS using CloudWatch Container Insights - Collect GPU metrics (utilization, memory, power draw, temperature) with Nvidia DCGM exporter - Visualize infrastructure and inference performance using Amazon Managed Prometheus and Grafana - Leverage community-driven Grafana dashboards for zero-configuration monitoring Inference Performance Metrics - Track VLLM (open-source LLM serving tool) performance metrics - Monitor time-to-first-token and end-to-end request latency - Analyze token consumption and generation patterns Agent Observability Beyond Agent Core - Instrument agents deployed on EKS (outside Bedrock Agent Core runtime) using OpenTelemetry - Enable auto-instrumentation without code changes using AWS Distro for OpenTelemetry - Configure telemetry collection through environment variables in Kubernetes deployments - Use CloudWatch GenAI Agent Core observability capabilities for agents on any platform Session and Trace Management - View complete traces with timeline and trajectory visualizations - Track token counts (input/output) for each model invocation - Analyze agent reasoning, tool calls, and decision flows - Correlate traces across multiple agent interactions using session IDs - Access integrated logs, metrics, and traces in CloudWatch Key Takeaways - Zero infrastructure overhead for observability setup - OpenTelemetry-based approach works with agents on EKS, ECS, EC2, or any platform - CloudWatch GenAI observability features extend beyond Bedrock Agent Core runtime For more events like this see our Cloud Operations Enablement series: https://aws-experience.com/amer/smb/events/series/Cloud-Operations-Enablement