Building Observable Systems with eBPF and Linux (with Mohammed Aboullaite)

Description

How do you monitor distributed systems that span dozens of microservices, multiple languages, and different databases? The old approach of gathering logs from different machines and recompiling apps with profiling flags doesn't scale when you're running thousands of servers. You need a unified strategy that works everywhere, on every component, in every language—and that means tackling the problem from the kernel level up. Mohammed Aboullaite is a backend engineer at Spotify, and he joins us to explore the latest in continuous profiling and observability using eBPF. We dive into how eBPF lets you programmatically peek into the Linux kernel without recompiling it, why companies like Google and Meta run profiling across their entire infrastructure, and how to manage the massive data volumes that continuous profiling generates. Mohammed walks through specific tools like Pyroscope, Pixie, and Parca, explains the security model of loading code into the kernel, and shares practical advice on overhead thresholds, storage strategies, and getting organizational buy-in for continuous profiling. Whether you're debugging performance issues, optimizing for scale, or just want to see what your code is really doing in production, this episode covers everything from packet filters to cultural changes in service of getting a clear view of your software when it hits production. --- Support Developer Voices on Patreon: https://patreon.com/DeveloperVoices Support Developer Voices on YouTube: https://www.youtube.com/@DeveloperVoices/join eBPF: https://ebpf.io/ Google-Wide Profiling Paper (2010): https://research.google.com/pubs/archive/36575.pdf Google pprof: https://github.com/google/pprof Continuous Profiling Tools: Pyroscope (Grafana): https://grafana.com/oss/pyroscope/ Pixie (CNCF): https://px.dev/ Parca: https://www.parca.dev/ Datadog Continuous Profiler: https://www.datadoghq.com/product/code-profiling/ Supporting Technologies: OpenTelemetry: https://opentelemetry.io/ Grafana: https://grafana.com/ New Relic: https://newrelic.com/ Envoy Proxy: https://www.envoyproxy.io/ Spring Cloud Sleuth: https://spring.io/projects/spring-cloud-sleuth Mohammed Aboullaite: LinkedIn: https://www.linkedin.com/in/aboullaite/ GitHub: https://github.com/aboullaite Website: http://aboullaite.me Twitter/X: https://twitter.com/laytoun Kris on Bluesky: https://bsky.app/profile/krisajenkins.bsky.social Kris on Mastodon: http://mastodon.social/@krisajenkins Kris on LinkedIn: https://www.linkedin.com/in/krisjenkins/ --- 0:00 Intro 3:01 The Evolution of System Monitoring 9:26 The Challenge of Continuous Profiling 13:40 What Is eBPF? 19:20 The eBPF Verifier and Safety 28:38 eBPF vs Traditional Profiling Approaches 32:18 What's the Overhead of Continuous Profiling? 36:41 Profiling Tools: Parca and Pixie 40:08 Managing the Volume of Profiling Data 47:57 Flame Graphs and Visualization Tools 49:35 The Three Pillars of Observability 56:30 Distributed Tracing with Session IDs 1:03:00 Getting Buy-In for New Monitoring Tools 1:06:25 Which Tools Should You Choose? 1:10:24 Outro