Design Foundational Data Engineering Observability – PyCon Taiwan 2025

Description

PyCon Taiwan 2025｜Day 2, R2 11:30–12:00 🪄 說明 Description 🪄 Data pipelines face unique challenges, often failing silently or suffering unnoticed data quality degradation that impacts downstream analytics, ML models, and business decisions. Standard application monitoring falls short. Foundational data engineering observability requires a dedicated approach, monitoring system health (logs, metrics, traces), data pipeline jobs and data-centric viewpoints. This talk introduces the essential data pillars – Freshness, Volume, Distribution, Schema, and Lineage – and explores practical Python implementation approaches. I'll introduce foundational techniques using libraries like OpenTelemetry, data quality tools (e.g., Great Expectations, dbt test), and custom scripts/metrics to establish baseline monitoring. Building this solid foundation is the critical first step towards enabling the advanced data-driven insights and deep correlation associated with the next version of observability for data pipelines. Slides: https://speakerdeck.com/sucitw/design-foundational-data-engineering-observability https://tw.pycon.org/2025/en-us/conference/talk/339 🚀 講者介紹 About Speaker - Shuhsi Lin 🚀 Focused on creating scalable and resilient data systems while cultivating a robust engineering culture. Expertise in guiding high-performance teams ensures the delivery of impactful data solutions, consistently applying DataOps principles. Currently, the focus is on elevating developer experience in the smart manufacturing and AI domain. Follow “PyCon Taiwan” ⭐️ Official Website: https://tw.pycon.org ⭐️ Facebook: https://www.facebook.com/pycontw ⭐️ Instagram: https://www.instagram.com/pycontw ⭐️ Twitter: https://twitter.com/PyConTW ⭐️ LinkedIn: https://www.linkedin.com/company/pycontw ⭐️ Blogger: https://conf.python.tw/