So what is open telemetry? Open

telemetry is a collection of APIs, SDKs

and tools and we use it to instrument,

generate, collect and export telemetry

data. Think of logs, metrics and traces.

In the past we had different tools for

different applications, different types

of servers and operating systems.

Various different tools to create and

generate logs, tools for collecting

those logs. Think of things like fluent

bit fluent D and then for metrics we

have Prometheus and then for traces we

have Jerger. Open telemetry looks to

consolidate all of this under one

framework. So what is open telemetry?

Well, it is a framework and toolkit that

is designed to generate, collect,

receive, process as well as export

telemetry data. And this telemetry data

can be anything from traces, metrics and

logs. The cool thing about open

telemetry is that it's an open standard,

an open framework that allows vendors to

use the framework. And this means that

you own the data you generate and

there's no vendor lockin. In the recent

months, I've started taking a look at

lot of different monitoring and

observability tools, some that are

managed and self-hosted. And one huge

benefit of open telemetry I found so far

is that you can run your own collectors.

Whether you're using Kubernetes or

virtual machines, you run your own

collector and you decide how to get the

telemetry data. Either you use a pool or

a push model. You can then process and

transform those metrics. Think about

things like Kubernetes clusters, you

want pod names, deployment names added

to your telemetry data. If you're

running in the cloud like EC2 instances,

you might want EC2 instance names or

specific cloud properties. You can

transform that data. You can also

process the data. Think about dropping

noise like logs and traces that contains

health probes. This allows you to

process and filter the data and then you

can forward and pass it on. And this is

the thing I found super powerful. This

means I can forward my data to any kind

of provider, whether it's a managed or

another self-hosted open-source product.

Think of any vendor that supports open

telemetry. This means I can quickly

switch between monitoring platforms to

try them out. A little while ago, I

discussed how open telemetry works with

the focus on logs. In this video, we're

going to be taking a look at tracing.

So, if you're new to open telemetry, I

highly recommend you check out how open

telemetry works with logging. A lot of

the concepts and standards are the same

when it comes to tracing and metrics.

So, in this video, we'll take a look at

what the collector looks like, how to

generate traces, and how to collect

them, how to process them, and forward

them onto a backend. So without further

ado, let's go.

Now I like to basically view open

telemetry as three as three main steps.

The first step is to collect then

process and finally export. Whether

you're running open telemetry on VMs,

containers or Kubernetes, these are the

three main steps you need to be thinking

of. How do we collect? How do we

process? And how do we export? But to do

all of this, we need an open telemetry

collector. Now, the easiest way to get

an open telemetry collector up and

running is to use something like Docker

or a container. In this example, I have

a Docker compose file. I create a small

network called tracing so I can run all

of these example apps on the same

network. Then I create a service called

OTL collector. I give the container a

name called Otel Collector. and I run

the open telemetry collector docker

image and this is enough to get a simple

open telemetry collector up and running.

The main thing is I'm giving it a volume

here called dot data and this volume is

purely for testing purposes because what

I want to do is I want to test my

collection the pulling of logs or the

scraping of metrics or the receiving of

traces. I may want to process it and I

want to test whether the data has been

received. So I can use a local file

exporter to do debugging. I always

recommend this when you're

troubleshooting collection. We'll take a

look at this in a second. Next up, I

have a config yaml. The open telemetry

collector is powered by a configuration

file and it's expected in this location

for this container image. And all the

magic happens in this config file. When

working with open telemetry, it's

important to understand the

configuration file because that is where

all the magic happens. And then you'll

start understanding those three

important steps. The first step being

the collection step. How to fetch things

and those things can be logs, metrics or

traces. And remember, open telemetry can

either go and pull or data can be pushed

to the collector. So it's very flexible.

In order to do this, we need to set up a

configuration file and we need to create

what's called a receiver.

[Music]

Now the open telemetry configuration

especially when it comes from the

documentation point of view can be

really overwhelming because open

telemetry uses a lot of terminology. You

have to get to grips with the terms.

They use terminologies like receivers,

extensions services pipelines

exporters, signals, baggage and all of

these kind of terms. And what do they

mean? So step one for us is to collect

telemetry in our case traces. And we do

this with the receiver. So the receiver

allows us to fetch things. If you go to

the configuration, you'll find the

config structure is explained over here.

We have receivers, processes, exporters,

connectors, and there are a lot more

terms. Go ahead and click on receivers.

Receivers allow us to collect telemetry

data from one or more sources. And they

can be pull or pushed base. So we can

use receivers to get data from

Prometheus, to get logs from pretty much

anywhere or to collect traces. We can

have another collector like another open

telemetry collector push traces to us or

we can have applications pushing traces

to us. With tracing, when you're running

in something like a Kubernetes cluster,

you generally set up a collector and you

use the open telemetry auto

instrumentation resource that you can

deploy in various namespaces and they

can target applications and those

applications or pods will automatically

be configured to start sending traces to

our collector. This is the best way to

collect traces within a Kubernetes

cluster. Let me know down in the

comments below if you want to see open

telemetry on Kubernetes. When you're

running in Docker Compose or in virtual

machines, you want to set up a collector

and you want to go and configure your

services to send metrics to you. So it

works a little bit different than

logging. So to receive traces, we want

to set up a receiver and under the

receiver, we want to set up a receiver

called OTLP. This is the open telemetry

protocol for tracing. And generally

traces are received over either gRPC.

GRRPC is a lot better for performance.

Most tracing clients and microservices

will use gRPC. It's the general default

one for open telemetry. For clients that

don't support gRPC, they can also use

HTTP. So generally for an open telemetry

collector that receives traces, we want

this entire receiver over here. We want

to set up a receiver that can receive

both gRPC and HTTP. And notice the port

numbers. 4317 is generally used for gRPC

port and 4318 is generally used for the

HTTP port. And we'll have these two

ports open on our collector. So this is

where we jump into that config.yml that

we're mounting in the docker compose

file. We create a receiver block like

this. The receiver block tells open

telemetry how to receive stuff. In this

case, we're receiving OTLP. We're using

the OTLP plug-in and we're exposing

these two endpoints. One for gRPC, one

for HTTP. Remember in our previous video

on logging, we talked about these

different types of receivers that are

available. These are basically plugins

that Open Telemetry supports. If we take

a look at the receiver from the previous

video where we covered logging, we had a

file log receiver that was able to pull

logs from Docker containers running on

the same virtual machine. So you should

get the idea that receivers are just

plugins for open telemetry that are

configured to be able to pull data or

receive data. Now that we have a

receiver configured, we can receive

traces. But what do we do with the

telemetry or the traces in this case?

Generally, open telemetry collectors

will want to send that to a storage back

end. But before doing so, we may need to

process the telemetry.

[Music]

This is where another configuration

option comes in which is called

processors. Processors take the data

that's collected by receivers and then

modify or transform it before sending it

to exporters. This can be useful for a

number of reasons. You may want to

enrich your telemetry with Kubernetes

metadata like pod names, container

names. You may want to filter out noise.

You may have a lot of applications where

you don't want to collect the logs. Or

you may have health probes in your

traces that you don't want to store. In

my open telemetry configuration, I have

a processes section. And as a best

practice, I have the memory limiter

processor which allows us to limit the

percentage of memory used by the

collector. This is useful if you're

running in a container with memory

limits or you're running in an

environment like Kubernetes with pod

limits or requests. This allows your

collector to run within a certain

percentage of that memory allocated and

prevents your collector from running out

of memory. And then I have a batching

processor. Basically, this just tells

the collector to use batching for

sending the data to exporters. The

larger the batches, the bigger the

payloads are going to be going across

the network. So some basic batching

configuration. And then the last

processor I have here is just a trace

rejection filter and this is just to

remove any health spans like any

endpoints or traces where you have like

health probes. This allows me to drop

health probes from my traces. And you

can also use this to optimize cost so

that you reduce the noise and you only

send the traces to your storage backend

that you care about. So now we have a

receiver defining how we want to get our

traces. We have a processor about what

we're going to do with the traces. But

these two configuration sections is not

going to do anything with the collector.

This is where the third construct comes

in

which is called a service. A service is

a key configuration for open telemetry.

The service section is used to configure

what components are enabled in the

collector based on other configuration

found in the receivers, processes,

exporters and extensions section. So the

service allows us to basically bring it

all together. As I showcase here, the

service is what enables the receivers.

It basically stitches it all together

using a few other subcomponents. So we

have an extension. In our case, we don't

have any extensions for our tracing.

Extensions is something we used in

logging. So I won't be covering that

here. But the pipeline is the key

section because with pipelines we can

enable tracing, logging and metrics. So

pipelines is basically just a group of

resources. So pipelines subsection is

where the pipelines are configured. In

our case, we're building a tracing

pipeline. So we can have traces, we can

have metrics, and we can have logs. In

the previous video, we covered logs. We

built a log pipeline. Now, we're going

to build a trace pipeline. Trace

pipeline will consist of receivers,

processes, and exporters. The service

section will hold the pipeline and any

kind of extensions we want to add on

top. So, coming back to the conceptual

configuration under pipeline traces,

we're going to bring it all together.

We're going to define our receiver and

that points to our TLP receiver we have

at the top here. We're going to define

our processor section. And that's going

to point to our three processors we have

here. Memory limiter, batch, and filter.

And we're going to define our exporters

here. Now exporters is the third main

key step with open telemetry. What to do

with the telemetry? We've received it.

We've processed it. Now, what do we do

with it? Exporters allows us to send

this telemetry to a backend. We can send

logs to things like elastic search. We

can send it to third-party monitoring

platforms. We can send it up to cloud

providers. For traces, we can actually

go ahead and store it in something like

a graphana tempo database. We can also

use exporters for debugging purposes. We

can write these traces to a file. We can

write it to the terminal. This allows us

to debug and test things. You can have

multiple exporters. So, you can actually

fan out to multiple monitoring systems.

This allows you to test out another

third-party monitoring solution if you

wanted to. And this is the key power of

open telemetry collectors. So, going

back to my open telemetry configuration

file. We've defined our receivers and

our processes. Now, we have exporters.

What I'm going to do is I'm going to

have two exporters defined here.

Actually, three. The debug one I'm just

going to leave empty like this. But the

debug one will basically print things to

the terminal within the collector

standard out. So, we can go ahead and

debug it. This helps us check whether

traces are being received. We can also

use the file exporter which just dumps

things to that volume I created. So we

can check the contents of the traces and

we can identify if we are receiving

traces from certain clients. This helps

us troubleshoot when we have three

clients and one of them don't work. We

can see what data we're receiving. Also

very good for troubleshooting. And then

the third one here I'm going to have is

an OTLP exporter. And just so you know,

you can actually structure these

exporters based on their name. You can

have the type of exporter like this is

file, this is OTLP, and this is debug.

You can actually give them a name by

putting a forward slash with some name.

This means you can have multiple file

exporters with different names because

one powerful feature of open telemetry

collection is that you can have multiple

OTLP exporters. You can send things to

tempo, you can send it to third party

systems and they all might use OTLP as

the exporter. So this allows you to put

different OTLP exporters. You can go

OTLP other and then send it somewhere

else to another hotel collector. This is

very useful because you can try out

other monitoring systems. You can also

have a collector per namespace in a

cluster if you wanted to break it apart

and have these all fan out to other

collectors. So you can have an open

telemetry collector like this one send

data to another open telemetry

collector. That's all up to you. It's a

very modular design. So in this example

here, I'm just going to forward it to a

tempo database that I'm running in

Docker Compose. So that's how you define

a bunch of exporters. And then we bring

it all together with a service pipeline.

We're going to enable a trace pipeline.

And we say here that we want receivers,

processes, and exporters. You have to

have these three sections. And this is

where you stitch things up. So my

receiver is going to be the OTLP one,

the one I defined up here. That's the

one over there. The processor is going

to be my three processors. my memory

limiter, my batch and my cascading

filter like so. So I have my three

processors defined there and my

exporters which are the exporters we

have looked at over here. And that is

pretty much it for the open telemetry

configuration file. And this is the same

thing if you're running in something

like Kubernetes. Learning this is

essential when we take a look at open

telemetry operator in Kubernetes. You're

going to be writing these types of

configuration if you're building your

own collectors. Now to bring this all

together, I have my open telemetry

collector ready to go. I've also

developed a bunch of microservices. Some

are just web-based front-end UIs, some

are backend.net, and some are backend Go

APIs. And what I've done, I've used the

open telemetry automatic instrumentation

libraries. You can find these on the

open telemetry website. It's a way for

developers to do zero code

instrumentations. In my docker compos

file, I have an example for Go and C.

Zero code instrumentation is available

for multiple programming languages. If

you're using the open telemetry operator

on top of Kubernetes, you can actually

automate this and have this

instrumentation injected automatically

into pods. Let me know down in the

comments below if you'd like to see that

on Kubernetes. So to show you that in

action, I have a bunch of microservices.

A video's web front end, a playlist API

backend, a playlist API hotel. This

container is needed because I'm

basically showcasing a sidec car. This

is something that open telemetry

operator does for us when we deploy to

Kubernetes and this only applies to go.

Here I have another videos API. This is

a car sharp application. Since the

playlist API is written in Go, we need

the sidecar container and this is how we

would export the zero auto

instrumentation to our open telemetry

collector. If you're using the open

telemetry operator on Kubernetes, this

is fully automated for net. This is a

little bit easier. I have a videos API

over here and all I need to do here is

define a couple of environment

variables. But here you can see I'm

pointing to my open telemetry collector

and then I have a couple of databases

videos database and a playlist database.

And these are all reddus databases. So

all these microservices will send traces

off to our open telemetry collector. And

then just to showcase that I have a

graphana instance here and the graphana

instance will use this tempo as a data

source. And this is where we will be

sending all our telemetry to. I can head

over to the terminal and I can do docker

compose up to bring everything up and

running. With the containers up and

running, I can head over to localhost in

the browser. Refresh that a couple of

times. And this is our video's web

interface calling a playlist API which

calls video API behind the scenes. So

basically demonstrating a fan out with

different programming languages and

Reddus database backends. It's a nice

way to illustrate open telemetry

tracing. And if I go to localhost 3000,

open the graphana instance. Username and

password is just adminadmin. I can head

over to the tempo data source and I can

hit the explore button. And then if I

give it enough time, I can see my traces

are starting to come through and I can

go ahead and select one of the traces

from my playlist API. And then I can

start exploring. If you want to review

the source code for this, you can find

it on the GitHub repo in the link down

below. In the GitHub repo, you'll find a

monitoring folder. In the monitoring

folder, you'll find a folder on logging,

which covers open telemetry for logging.

You'll also find a folder for tracing,

which also has a readme file and a

docker compose file that we ran today.

In this tracing open telemetry folder,

you'll find all the applications that

we've taken a look at today, the config

for open telemetry collector, as well as

the docker compose file to get

everything up and running locally with

ease. So you can easily go ahead and get

this up and running with ease. So

hopefully that helps you understand how

tracing works in open telemetry. And if

you want to see more in the realm of

tracing and monitoring, let me know in

the comments down below. And if you like

the video, be sure to like, subscribe,

hit the bell so you know when I upload

next. And if you want to support the

channel even further, be sure to hit the

join button down below to become a

YouTube member. And as always, thanks

for watching and until next time.

Peace.

[Music]