LLM Black Box: End-to-End LLM Observability with Datadog & Google Vertex AI

Description

🎯 Hackathon Project: AI Partner Catalyst - Datadog Challenge LLM Black Box is a demonstration project for implementing observability in LLM applications. Built and tested in a local/simulated environment to prove end-to-end monitoring concepts. The Problem LLMs need specialized monitoring that traditional tools don't provide: - Tracking token usage (cost control) - Detecting safety violations - Monitoring performance and reliability Our Solution (Proof of Concept) A test application demonstrating how to: 1. Use Google Vertex AI Gemini 1.5 Pro 2. Instrument with OpenTelemetry 3. Stream LLM metrics to Datadog 4. Create detection rules for common LLM issues 5. Test with simulated traffic patterns 🔬 What We Built ✅ Local FastAPI app with Vertex AI integration ✅ Datadog Agent in Docker for telemetry collection ✅ Custom metrics capturing tokens, safety ratings, latency ✅ 4 Datadog detection rules (anomaly, safety, latency, cost) ✅ Dashboard visualization of LLM metrics ✅ Incident automation when rules trigger ✅ Traffic generator for testing scenarios 📌 Project Status This is a hackathon demonstration project tested in a simulated environment. It shows production deployment patterns but is not yet deployed to production cloud services. 🔗 Resources GitHub: https://github.com/aurshitha/LLM-Black-Box License: MIT - Free to use and extend Built for: AI Partner Catalyst Hackathon - Datadog Challenge