In this hands-on Kubernetes operator

course, you'll learn how to extend

Kubernetes by building your own custom

operators and controllers from scratch.

You'll go beyond simply using Kubernetes

and start treating it as a software

development kit. You'll learn how to

build a realworld operator that manages

AWS EC2 instances directly from

Kubernetes covering everything from the

internal architecture of informers and

caches to advanced concepts like

finalizers and item potency.

Shubhamqatara developed this course.

>> Now if you already know Kubernetes, you

know that there are concepts and

Kubernetes objects like pods,

deployments, replica sets, stateful

sets, services and so on so forth. But

do you know that you can create an

object called EC2 instance?

No. Well, that's the beauty of

Kubernetes because you can extend the

current capabilities of Kubernetes and

create something which is called an

operator. So you can create an operator

to control things which are outside of

Kubernetes like EC2 instance which we

will learn in this particular course.

I'm very excited to bring you the

Kubernetes operator course from scratch.

This 6R plus course is brought to you by

Shouhham who has 8 plus years of

experience and working in Tago have

trained many on open shift holds

multiple certifications like GCP cloud

professional and devops and this course

comes as an outcome of his work at

Privago for building custom operators in

production. Yes, we'll build a

full-fledged working operator end to end

from scratch learning why it is even

important, how to do and everything

about cube builder and then building it

end to end. I'm really really excited

about this course and cannot wait for

you to get started. So before we can

build a custom operator for Kubernetes,

we need to know what is an operator,

right? And before that there is a term

that is called a controller that you

really need to be familiar with. Now

many of you might not might know already

what a controller is. It's you know um

you have heard about this which is the

cube controller

um manager.

But what does it really do? What work is

it that the that the controller is is

responsible for?

So a controller is nothing but it is

think of that as a forever running loop

right think of this as a piece of

software which we will be writing that

is a forever running loop and if I want

to write a bit of pseudo code for that

it's kind of like this so you always run

it and the first thing that it does is

it um it observes the state of the

resource whichever resource you are

writing an operator for you will have a

controller for that as well. So if you

want to work with pods or deployment you

want to work with services you want to

work with config map there is a operator

for all of those resources. So the first

thing that it does is it keeps on

observing the state of your resource. If

the state is updated

again for whatever reason you updated

the state, maybe in your deployment you

change the image, maybe in your config

map you edit the data of the config map.

Whatever reason that happens and if the

resource is updated, the second thing

that a controller manager or an operator

or a controller really does is it

compares the current state to the

desired state. And this is where you put

your business logic. This is actually

where you define what to do in case

there is a drift that is recognized and

most importantly what not to do if there

is no drift because it's very important

to make your operators or at least your

controllers uh ident.

They have to be amputent. I cannot

stress this enough. We will talk about

the reconciled loop just in a minute.

But this has to be important in terms of

if in case your resource needs no

change, there should be nothing done on

Kubernetes. You should be able to run

your controller as many times, but it

should not result into a change if there

was no change needed. And if it finds

that there's a drift between the current

state and the desired state, it then

does an update. Or you can also say it

acts on what logic you have asked it to

do what to do in case there was a drift

found. And then uh we close this. So

it's a forever running loop that never

stops and keeps on watching the API

server for your resources that you are

managing.

Now what we are going to build is a

cloud uh it's a cloud controller because

what we are building will be a piece of

software that actually runs on your

Kubernetes environment here. Let's say

this was your Kubernetes and there you

say I want to make kind EC2 instance.

Let's put it this way. It goes to

Amazon, sees if this instance with this

name is already there or not. If it is

there, it does nothing. If it's not

there, it creates something. So, it's

kind of what we would call a cloud

controller. Think about when you run on

EKS, when you go to Azure Kubernetes

service, it is very easy for you to

change the service definition, the SVC

for example in in EKS to have a load

balancer. You can just say the type of

services load balancer and in your EKS

cluster there is a software which is

working which is running that abstracts

how to create a load balancer how to

make your service as you know as

backends of that load of that load

balancer it hides away the complexity

for you and that is what a cloud

controller does. There may be many

different controllers that cloud

providers will give you in their own EC2

in their own u kubernetes distributions

to make your lives easier so that you do

not have to know the the nitty-g

gritties of it. You just say I want a

resource and then you get one and that

is what a cloud controller manager would

be.

Now when I was talking about uh

controller we spoke of this term called

ident and this is something I actually

want to um and you know um explore a

little bit with you. So there are few

things that your code should actually be

doing when you write a controller when

you write u the logic for what to do

there are some things that can actually

be uh that can actually be done. Um and

the first thing is a happy path. So what

is a happy path? Um you have a logic

your reconcile you know this is actually

also called as reconcile loop. It's

here. This drives the cluster state to

your desired state and this is what it

reconciles and that is why cubernetes is

eventually consistent. And I mean in a

way that you make a change eventually

which is a very short time again that's

why we don't we think it's this but

eventually your state is going to match

the desired state that you want to do

the the cluster of state is going to

match the desired state.

Now let's zoom in in this path a little

bit where we have our um you know um

case one where you have a logic your

resource got updated and your reconcile

function is then triggered. This is

where you know this is the start of your

uh loop. Let's put it this way. This is

the beginning of your loop. So the first

thing that you do is uh you get your

object from the request. The way it

works is when you update a resource in

Kubernetes and there's a loop that's

watching on that, there's a controller

that watches on that. The controller

gets a request. The controller can

actually get the request that you

wanted, which is the API request to the

API server and it can get the object

data. For example, if you updated a

config map, your reconciliation loop,

your reconciliation loop can actually

get the YAML or the JSON of that config

map. So you can verify or you can

actually you know um see what has been

changed or what updates has been done,

what has been done by the user on that.

So you can get the object from that

resource and you can then observe the

desired state from the spec. What you

actually do is you see um you define

your uh config map for example or let's

say a pod. So you have a pod and then

you have a dospec in which you define

your containers.

So you can see how many or what spec is

there for a particular resource and then

you can compare uh with that spec what

is the actual state of the resource if

they match you know if the if the number

of containers in your pod are exactly

what you wanted then you have to uh you

have to just you know skip it you don't

have to do anything and this is what the

happy path is you do nothing and this is

absolut absolutely important that you

realize you don't have to do anything in

this case. You don't have to make any

API calls. You just ignore that request

to your reconciliation loop because the

actual state is equal to the desired

state. And that's what happens when you

exit your loop gracefully. Of course,

I'm not saying you will stop the loop

because you have to keep on listening on

the request, but you will not make any

changes.

There's also a second thing that can

happen. So uh in this step you have your

function triggered. You get the actual

object from the request. You see what is

the spec of the object. What object is

being modified and what is the actual

resource of the of the actual state of

the resource. And this is where it gets

interesting.

If the desired state is equal to the

actual state that you want, you do

nothing. We know about this from the

previous happy loop. However, if they do

not match, for example, in your

deployment, the current that you have in

HCD is your replica

three. Let's take this example. This is

nice. So let's say uh your current uh

one that is there is replica equal to

three and this is for a deployment which

is stored in HCD. This key is stored in

HCD. Now you do um a cubectl edit you

know you do a deployment and then you

give the name of the deployment and then

you save that file. First thing that

happens is that your reconciliation loop

will get this request that okay because

I'm watching deployment this deployment

is now updated and that is where you

made the change to be replica equal to

five.

What your current is three your desired

is five. Now you say okay in your spec

you will have replica equal to five.

This is what you can do when I say

observe the desired state. You get the

actual object YAML. You get the actual

object YAML and then you observe uh this

desired state. So you want five replicas

and you observe the actual state which

is still replica equal to three. So

there is now a drift. the current

actually does not match the desired

state and this is where your logic would

actually come into the picture what to

do in case your resources are not

matching to what the user has actually

asked to do. So there you will calculate

some differences. You will probably take

some actions. You will do a create,

update, delete for the resource. In this

case, you will create five more pods.

Sorry, you would create two more boards

because you wanted five. So 3 + 2 is

going to be five pods, which is actual

user uh requirement. And then there if

your action is succeeded, you update the

status field. And then you exit the loop

again. This is very important you know

uh every every resource in Kubernetes

has a dot status. So you have a spec and

then you have a status and this is how

the reconciliation loop knows if it is

actually matching. If for some reason

you could not create the pod for

whatever reason it may be you can return

an error and then you can reue retry

doing that action. And this is what

makes Kubernetes as healing. It tries

again. It tries again with a you know

with a back off. You can configure this

that if you were not able to do this

right now maybe there was no uh let's

say you could not create the pod because

you did not have enough memory. Your pod

would actually not be created or they

actually put in the pending state. This

is not a good example but let's say for

whatever reason your pods could not be

created. Maybe you were missing the role

based access control in this name space

where the pod should be created. Now it

will be recued and the way it goes is it

goes back to the beginning of the

reconciliation loop here and then it is

started again and this is what happens

when I say you need to recue. Recue

means you retry that action and this is

what Kubernetes is about self-healing

because if you give the rolebased access

control to the you know to to the

controller it will be able to create

resources. It's not like I tried once

and I couldn't do it. It keeps on uh you

know uh trying again and again. You

might have seen this. If you have a pod

which needs a persistent volume, um it

goes into pending if there is no P lab

that the pod needs. But if you create

one, the pod automatically gets

scheduled. It gets started. You do not

have to do that. And this is the beauty

of the loop that can reue that you can

recue for your um for your cases. And

this is this is absolutely the

brilliance of um self-healing

in Kubernetes.

Now one thing you have to be very

careful is this. There's also a sad path

and this is something you always always

want to avoid when you are writing a

custom uh controller.

The things are pretty much the same. So

what you do is you start your loop. You

got a request that somebody updated the

deployment. You see what they have made

the changes to. You see if there's is

actually there or not. The change there

is if actually if you go to the desired

state you have to absolutely

do nothing. You have to do nothing.

Absolutely nothing. What I mean by that

is you do not have to update the

resource for anything because when you

update the resource let's say here you

know um let me talk about that.

Now this is interesting. The way it

works is I'll go back to the the one

where you had to make some work you

calculate uh the difference and you

update your resource. Now if that action

is succeeded if that action is succeeded

you will actually be triggering because

you updated the resource. This will

actually trigger the reconcile uh

feature again.

Kubernetes controllers they do not know

what you have updated whether you

updated the spec whether you updated the

metadata whether you updated the status

they don't know about that they just say

okay the resource deployment was updated

here so I will re retry my actual you

know I I would rerun this from the

consiliation loop and now because you

updated because you created five pods

now your replica

is actually five and it will now say

okay um I get the object the replica is

five that the user wanted and now

because I'm running this again the

replicas have been already created the

state matches I don't have to do

anything you have to write your

reconciliation boobs ident

maybe you uh You got a request and uh

your object you get the object you

observe them they do not actually make

need a diff they don't need any work

maybe you have got the same replica was

five and then your actual state was also

five you do not need anything but

by mistake you update the status of last

sync you say okay it's just the metadata

it does not change my deployment right

it doesn't change my containers. It

doesn't change the image I'm using in

the environment variables. It doesn't

change that. I'm just putting

as a good person. I want to see when

this was last synchronized. And you say

that whenever a request comes, even if I

make no changes, um I would update the

the status.los sync, which actually

would then trigger an API call. And you

see whenever you update your resource it

goes back to the beginning of the

reconciliation loop and this is where

you would have a forever running loop

request comes in. Um okay you got the

object you observe the desired state

from the spec. I'll zoom in a little

bit. uh there was actually no need of

any changes on the resource but you by

mistake you are updating the status. So

Kubernetes says okay the object the

controller is looking for has been

updated. So it goes back up to the

beginning of the loop and then you

update the sync uh the last sync again.

Kubernetes says I got a new update from

the beginning and this loop will

continue forever. your resource will

keep on updating without having any you

know without any stock. So this is very

very important that you need to be very

careful of um not making any changes if

you do not require any changes.

Now this is the foundation this is

actually the foundation of uh how to

write a operator how to write an

operator. The controller is the actual

logic that you have to have. Now what uh

and there are a couple of things uh when

you are writing an operator this is

absolutely important I think this is a

good thing you should read this the most

important question you should be asking

or your controller should be asking is

if there's anything for me to do that

means if the current state is equal to

the desired state if not exit

immediately do not do anything there

should be a golden rule as well that you

should follow that you should only write

to the API server when the actual state

differs from the desired state. Where in

this case you see here you you are like

okay I know that the actual state is

equal to desired state. I make no calls

to the API server. I do not update my

resource. But by mistake you update the

last sync which is again a request to

the API server to modify the resource.

And then the reconciliation loop sees ah

there's an update. let me go back and I

would restart that uh I I would rerun

the loop and then it's a problem. So you

always um have to make sure that you

only make the changes to the resource

when they differ from the desired set.

And this is also what a tempotent means

that you can run your loop 100 times if

the cluster if the machine is already in

that state you should not be doing

anything. you know it doesn't break

anything it doesn't change anything if

the cluster state is equal to the

desired state that is absolutely

important to uh to be to be taken into

account and this is what's interesting

this is what makes these operators uh

resilient which is they are stateless

they don't remember what they did with

your resource in the last request they

they don't do that they don't remember

they don't remember if the Paul replica

was three or five or seven. They don't

remember if you have the environment

variable or not. They always always

check the resources. They always their

source of truth if they go to the

required you know place maybe you're

writing a a cloud operator they go to

the cloud maybe you're writing a

database operator which creates a

database it goes to the database always

runs the query and this is why uh these

are stateless. So your container your

your controller can actually be killed

or the you know the node on which it was

running it could be deleted it could

crash the container uh the controller

will go to another node it just starts

from there it doesn't have to have a

persistent volume to store the state it

doesn't know that and this is why it can

crash restart and still figure out if it

needs to do something uh on a particular

resource or not because that's what you

have made it to do you have the logic

that it always observes, it always

checks the desired state in the current

state and if there's anything to be done

uh it does it otherwise it says cool the

uh resource is already in that uh uh in

that state which the user wanted me to

do.

Now if you talk about um uh this is

about controllers but what is an

operator?

I think you guys might already know

about operator in in a in a way because

you want to write your own operator but

let's just go through that quickly. Um,

imagine you guys want a house. You know,

you you get a house. Let's say you are

living in India. And this is a very good

example that I like. Let's say you are

living in India. You have a house

already. Maybe your parents own one. And

one day you decide to move to Germany.

The place is completely new to you. You

have never been to Germany before. You

don't speak the language as well. You

don't know German. Now you need a place

to stay. you need a house to stay. you

call a company uh you know in this case

you call a company and the company says

hello sir you're moving to Germany we

would make would help you make sure your

move is easy and simple we have two

options one we can give you a full

furnished house

we will give you a full furnished house

and also we have another option where

you can just get a simple uh unfernished

I'm saying a simple house but let's say

an unfernished house.

You can choose whichever you want and we

would be happy to give you the key when

you land in Germany once you sign the

forms and everything. The company also

says one thing that sir while we are

giving you the furnished house we also

give you a helper.

Now you say what is this helper? What is

it going to help me with? The company

says if at any point in time you break

uh you know a tap, maybe your water

filter is broken, maybe the floor is um

you know you spill something on the on

the carpet. Are you going to fix it?

Maybe your bathroom uh tap is broken.

Maybe you break um a window. You you

never know. You don't know anyone in

Germany. you will fix it yourself or you

can help you can have the helper do

these things for you because you don't

know the nitty-g gritties of where the

hardware store is, how to call someone

if I lose my keys for the house. Let the

helper do it for you. So the helper is

actually someone who has the full

knowledge of this house, who has the

full knowledge of how to fix things if

they goes wrong. You just have to tell

the helper maybe you lost your keys, you

know, just tell the helper, go get me a

key. He knows where the store is. He has

the logic. He has the knowledge of where

the store is. He has the knowledge of

where to go and in what language, how to

speak to the to the person who can make

you a key in German and gets you a key.

If you have a broken pipe horse, he

knows how to fix it. So, think of this

guy. Okay, this helper as the actual

operator. Now if you want to port this

in um in the terms of software, think

about you uh have a database which is

called MySQL. Now for you uh

installation of things is easy now

because you have a container you can

simply run it and you would be able to

get your app your software but what

about day- operations? What about maybe

you want to do a database migration of

your schema? Maybe you want to take a

backup. Maybe you want to take

incremental backups on a particular, you

know, a schedule. That knowledge needs

to be either with you or someone who can

do this for you. And this is where MySQL

not just gives you the database MySQL

but also has an operator for you. This

operator is actually a controller

running internally. So this controller

has all this logic. If the user asks me

to uh create a database, I know how to

make a database. I know how to log into

the DB, I know how to create a database,

I know how to do that. It knows about

it. So you just have to tell what to do.

In this case, this helper was the

operator. And in this case, this was the

MySQL database product we were actually

looking for. And that makes your life a

lot easier because you don't have to

worry about the lower level details. Now

operator

has two things. One is a custom resource

definition and then the other thing is a

custom resource.

You know how you can do cubectl get

pods. You get a response. Maybe you have

pods or not. It says yes I have pods or

it says no pod found in the name space.

But if you do cubectl get apple, it

doesn't know what this resource called

apple is because kubernetes has its own

vocabulary. It has the API resources

that it has been told to remember and

those are the resources the internal

ones that are native to Kubernetes like

pod deployment secrets uh services you

know um all these things these are

resources that Kubernetes knows about.

But what if you want to create your own

resource which is in our case what we

will do is going to be an EC2 uh

instance.

I could also want to create an S3

bucket. In that case, I need to expand

Kubernetes's vocabulary that okay, this

resource called EC2 instance. If

somebody says uh it gives you a YAML

which is kind uh EC2 instance, you know

how to create or at least you know what

that is. What to do on that? That's a

different story. You know what that is.

So that if somebody gives uh on this

file cubectl create you can don't just

tell me you don't know what is this

resource you know about that now I have

given you the schema of what an EC2

instance would be I have given you this

custom resource definition so whatever

the user gives you in this kind

acceptepic because now your vocabulary

has been increased and this is going to

be a custom resource whenever you create

uh whenever you instantiate a custom

resource definition that is called a

custom resource. For example, if you

created a custom resource definition uh

for EC2 instance when you create it and

then you can do cubectl get uh EC2

instance.

What you receive is an instantiation of

the of the definition that is a custom

resource on which very important on

which your operator your controller

will be acting upon. So your controller

knows that on a resource type EC2

instance it has been created it has been

deleted. If you create a resource called

EC2 instance, it knows that on this

resource there was an update which is to

create the resource. The controller will

create that resource for you. If you

delete that, the controller will say

okay on this resource which I am

watching there's a delete operation

performed by the user. So it goes ahead

and deletes it for you. So without the

controller your your custom resources

are nothing. They are just kubernetes

knows about it. It does not react on

that. It does not acknowledge that okay

I'm going to do what you want me to do

because it doesn't have the knowledge.

So while the CR and CRD uh you use them

to tell what you want the controller

with them is actually the how part of

it. How do I do that? And this is what

we going to be building. We will be

building um a cloud controller which is

for building EC2 instances on Amazon.

And this is what we will be looking for.

Um there's also something which you need

to know. Kubernetes is not just a

platform now. It is a complete operating

system for um you know for people. So

let's talk about how Kubernetes is

actually expandable and how can you use

Kubernetes as an SDK. So what's very

important with Kubernetes is to look it

from not just a platform where you can

run your applications but rather how can

you expand Kubernetes as a software

development kit and what can you do with

that on other platforms that's also what

you can do. So the first thing that

Kubernetes is so widely adopted by cloud

providers by onprem for other softwares

is because of its extensibility.

Get me let me get a color different. So

it is because of the extensibility

because of these custom resources

because of these operators and because

of the controllers and this is what uh

we just talked about. Kubernetes also

have API first approach. So everything

in Kubernetes has an API. Everything

your pod is an API. Your service is an

API. Your API server has APIs for all of

these things and that makes it very easy

to u write your code for and there are

client libraries for this and that makes

it very very easy. You have the SDKs

that you can build your controllers on

uh for Kubernetes. there is Go, uh,

Python, there's Java, there is

integration of JavaScript with

Kubernetes because there are client

libraries for that as well. And you can

also, uh, Kubernetes has backward

compatibility because it does not just

delete API resources, it deprecates them

first. It gives you enough time to move

towards a different um, uh, uh, you

know, to a different API um, version and

it also versions its API. So maybe you

might have seen uh pods/v1

or you might have seen network

um you know um network config

/v1

beta 1 beta 1. So this is the version of

kubernetes um API. So it makes it very

easy for you to develop new APIs without

breaking the existing ones and that

makes it really really simple or really

helpful I would not say simple but

helpful to expand your APIs and this is

a plug-in everything you can have your

networking you can bring your own CNI

you can choose from different CNIs quite

popular ones are stelium um I think yeah

selium is one very popular from isalent

which is a company acquired by Cisco. Uh

you also have different options for

storage. You have different options for

runtimes and web hooks where you can

intercept everything as a admission

controller which could either validate

your request or which can either mutate

your request. I think for these web

hooks we can have an entirely different

course for it. they deserve their own

time because I don't do justice if I

just talk about there is an admission

controller which can validate and uh

mutate it doesn't doesn't help so

probably something to look for in the

future for us

and this is why different cloud

providers because of this extensibility

of kubernetes there are different

flavors and thousand plus tools that you

can use on top of kubernetes so there is

open shift from red hack there is suz

from Rancher, Tanzu from WMware. Then

there's softwares on top of that which

is cubeflow K native um cube which is

also quite popular nowadays and that's

what makes the developers happy because

they say what not how. Now if you are

working in a platform engineering team

uh you want your let's say you know this

is a developer this developer wants a

machine in Amazon he wants or she wants

an EC2 instance and you manage your

cloud let's say you are the cloud uh

admin who will give them the EC2

instance they come to you you uh say

okay I run some commands blah blah blah

and this is the instance and you give

them that that's okay but this is a very

old approach.

What you can rather let these guys do

and this is what um in in internal

developer platforms would actually help

you with or you can build your own then

you can say okay listen what if you want

an EC2 instance you don't have to come

to me just give me this YAML which is

you know you can explain them explain it

to them you can have a Helm chart around

this that says I want an instance where

you can say the number of instances

maybe two the instance uh type which you

want and then maybe the you know you can

have them give the um the instance uh ID

where you can then say the machine the

AMI ID that you want to use a very

simple thing and then maybe also the

port numbers that should be open. They

give you this in a YAML format and you

pass this from your controller,

you know, after you can have a pull

request review. So after they have a

pull request, they this is stored in

GitHub. You have a pull request and then

they get an EC2 instance. With this,

they get to say what they want. They

don't care about how to create resources

in EC2. They don't care about BPCs. they

don't care about anything and also

because you have a githops workflow now

you can have argo cd uh deploying these

resources and then the controller takes

care of creating the e2 instance

everything is as a code you can have a

githubs very resources um very very

simply with this platform uh as as a you

know as a as a product which is platform

engineering all about so you can have

the declarative options you can use Helm

to help the lives of developers easy

that they can just give you this

information. You render the resource and

then your controller takes care of that

and and this is this is I cannot um

stress it enough how how simple it makes

our lives easier.

Now because you can run Kubernetes not

because the thing is you can run

Kubernetes anywhere and the reason why

you can run Kubernetes anywhere is

because of the standardization. You can

run this in any cloud you can run this

on edge you can run this you can run AI

workloads on top of that anywhere you

know Kubernetes is standard because it

has one pattern which is a controller

pattern that rules them all. Um I would

say DNS just works again it can be

problematic but every pod knows where

the where every other pod is. um it has

its own challenges depending upon how

many number of services you have in a

cluster, how many pods you have in a

cluster. Scalability could be another

issue but for a for a cluster that you

have bootstrapped, it just works fine.

And then you have config management for

your developers which I don't think I

need to uh talk about.

The single I'm trying to make here is

it's not just a container orchestrator.

It is a complete operating system. You

want networking, it has it. You want

memory management, it has that. You want

compute management, CPU, uh, storage, it

has that. It has disk management, it has

it. So, you can actually build and

package and ship your software that runs

on top of Kubernetes. Uh, any sort of

software that you can uh, you know, you

can build and run on top of Kubernetes.

It's not like you're just using

Kubernetes, but you can ex expand it

with all of these controllers and these

um operator frameworks that we are

talking about. And this is why I love

Kubernetes a lot. All right. So, this

was about how do you use Kubernetes as

an SDK. Now, let's talk about um how do

you bootstrap Kubernetes? Um how do you

bootstrap a Kubernetes operator with uh

with a software called Cube Builder? And

this is where our journey would be

beginning. So let's go on and do some

hands-on on writing an operator.

So before we can build our own

cubernetes operator, we need a place to

run this operator on and that is going

to be Kubernetes. Now you can build a

Kubernetes cluster in GKE. You could

probably use Amazon as a managed

service. You can build your own clusters

with uh QBDM. Whichever way you want to

do it is fine because the operator that

you are building it will be built into a

container image and that container image

can be run on any Kubernetes cluster. In

our case, we want to keep it simple. So

I'm going to be building the operator

and I'll be testing this operator which

is going to be running on my cluster

locally and create instances on Amazon

which is external to the cluster just to

show that you can manage infrastructure

that is external to your Kubernetes

environment. And this is why Kubernetes

is really popular because it lets you uh

use it as a SDK as an operating system

of the cloud which we will also talk

about in the future. So K3D is a

Kubernetes distribution by Rancher which

has many other distributions like K3S

which is also a very simple lightweight

Kubernetes distribution. It also has RE2

which is more hardened and for security

if you are working in the governance um

and K3D it lets you create containers or

rather it lets you create Kubernetes

clusters in containers. If you have kind

you can use kind. If you have K3D you

can use K3D. If you have a sandbox

cluster somewhere, you can use that as

well. The reason why I'm using this

local is because it's very lightweight.

It does not cost me lots of resources.

It's free of course and it's very fast

because it's running on my computer.

So for K3D, we can install that very

simply. Just go to the installation

script and you can download that with

either cur or you can download that with

wget. I would suggest you go with the

latest version. And once you have this

downloaded, you can do K3D or K3D

version. And I've got the latest version

of K3D, which is 5.8.3.

And the Kubernetes version that I would

be using when I build a cluster with K3D

is going to be 1.31.5.

But there is a newer version of

Kubernetes. What if I want to use that

better? We are DevOps engineers. We are

uh cloud engineers. We like to have a

single source of truth for all of our

applications which is why we do githops

right and wouldn't it be nice if you can

just version control your clusters as

well uh that right now I have got one

cluster which has two agents maybe I

want to increase it let me put into

GitHub and that is exactly what K3D

allows you to do with a very simple

cluster config file and this one has

lots of options which you can go to K3D

uh and look on the documentation.

However, I I've kept it very simple.

This one gives me one master. K3D allows

you to create multim masteraster

multi-node cluster. Again, I'm just

going with one because I don't need high

availability.

And second, I'm going to be using two

agents here, which is going to be the

worker nodes. And this bit tells me the

version of the Kubernetes that I want to

use. And that's the one which we will be

using.

You also need Docker because K3D uses

Docker because it creates containers in

which it runs your Kubernetes cluster

which runs containers and that's a whole

inception going on out there. But these

are the two things that that I would be

using. If you have any other

distribution of Kubernetes, you can very

simply use that. So I've got Docker um

running on my machine. I've got or stack

which is actually giving me docker in in

the background which is giving me a

runtime in the background I would say

and to talk about K3D its architecture

is fairly fairly simple. So what it does

is that this is how it looks like. So

you have your laptop or you have your

computer on which you want to create

multiple Kubernetes clusters. Now as a

developer I might need different

clusters for different applications. I

might want to promote them from dev,

testing, QA just to have a pipeline

going for a complete software

development life cycle. That's also

possible for me too. And that is where

K3D shines really well. When you make a

cluster in K3D, it creates a separate

Docker network for all of them. So they

are completely isolated from each other

and they have their own tier as as you

will. have their own network uh in which

they would be talking to. So here you

can see I've got one cluster here which

is blue and there's one cluster which is

green cluster A and cluster B and this

is the master node and these are just

robots which is our work is cuz that's

where the actual work gets done and we

have these docker networks created right

now if you do docker network list you

see the standard docker networks that

are created when you install docker

however when you do k3b cluster create

with this config file which is our

source of truth. When you do that,

there's going to be a new network

created which I just showed you. So we

will see that just in a moment. Once

this is created when you you know when

you ask it to create a cluster not just

it creates your cluster for you not just

it sets up a gateway for you not just it

creates your workers for you it also

updates the cube config or rather it can

help you to get the cube config and here

you can see my context is automatically

set to cubectl. It says you can use it

like ctl cluster info. And if I do that,

that's where my clusters are. That's

where my cluster is running. Now if you

do docker ps, you will see there are a

couple of containers that are just

started. And this is our infrastructure

for K3D. We have got two agents which is

our worker nodes. We have got one server

and we also have this engineext proxy

container which is there for some reason

and the reason why it is there is for

you to talk to your API server because

you can use K3D to create multiple

masters. You need to have a load

balancer. So you should not be needing

to set it. That's why K3D does it for

you. And here it creates a container

that is listening on your port on your

computer's port which is 5745

and that's actually uh forwarding the

traffic to 6443 of the master or in case

you have multiple of the masters and

that's why you see the Kubernetes

control plane is running on 5745

on all the IP addresses of your

computer. If you go to this port, you

will be talking to Kubernetes. You will

be talking to the cube API server.

Now, what can you do? Every time you

have a cluster, it's good to do a smoke

testing. A very simple one. So, we can

do cubectl

get nodes. There you go. You have got

one control plane, one master. You've

got two agents which are ready. You can

do cubectl get service. There you go.

You can do cubectl get pods and some of

them are code DNS which is very simple.

It comes with a metric server also. It

comes with traffic insert which is again

uh it allows you to expose your services

outside or work as an ingress if you

will. Um and it has got a local part

provisioner which is for storage. I

talked about the metric server already.

Now let's try to do some smoke tests.

And if you can do cubectl create

deployment or kc create deploy

it's going to be creating a deployment

and it's going to create a pod um k get

pods and here you can see it's container

creating. If I do k logs and if I can do

my deployment this is a log for

engineext. That is fairly fairly simple.

If you had used engine x this should be

nothing new. You can also expose your uh

deployment. We want to check the network

connectivity between our applications.

If one service or one pod can talk to

other application in the cluster, let's

just validate that. So I could do uh I

want to expose my service. I want to

expose my deployment called my

deployment and the port number for that

would be 80. Here you can see it's a

service resource in Kubernetes and it

has got a cluster IP. Now you know if I

want one application to talk to uh

another application in my Kubernetes

cluster I can use this cluster IP and

that's exactly what we would be doing.

What we would be doing is here okay um

so here we have a pod in our new cluster

for which we just created a service. I

want to test the networking in my K3D

cluster. So I would create a new pod. I

would try to curl this service and I

should get a response from this pod and

I should be able to curl it because it

is HTTP cuz I know I just ran an enginex

server and this should work because it

is a single cluster. You know you cannot

by default expose your service IP

addresses outside the cluster. However,

inside it should work fine.

And that is where we can use our trusty

curl image. This lets you just do a curl

to any other IP address or host name.

And that's where we can do k run. I want

to use this is my container. I want to

create a curl container with the name of

curl. This is my image. And I want to

connect on the IP address of my service.

That's that. Let's look at the pod. This

pod is container creating and it's

completed already. Crash loop back off.

That's fine. Let's check what happened.

And if I do logs for curl, it wasn't my

crash loop back off. It just started,

exited, started, exited, and it's like

what is going on? It was not a chron job

that runs till completion. Um but you

can see here this is the response that

you get from the service uh which is

engine X and that tells me that my

cluster is ready for connection. My

cluster is ready for me to build

applications and also uh you can

probably go to um you can also check

from your cluster if you have external

connectivity because we would be talking

to Amazon.

Might as well check that. So we can do k

run curl or let's say Google and I could

do httpswww.google.com.

Do I have a pod now? Uh, Google

container creating and let's say and

that looks Google to me. Um, looks fine,

right? So, we have connectivity between

our applications and we also have

connectivity now uh to external

environments and this is going to be the

foundation on which we will be building

our application.

Um you also would be needing to have go

on your computer which we talked about.

You need docker git the standard

developer tools. So um that's it. This

will be our uh our setup. Now I think we

should talk about what are you going to

be really building in this course and

what is a reconciliation loop? How does

kubernetes know what you want it to do?

How does the controller or what is even

a controller in the first place? How do

they know that I want to do something?

The user has asked me to do something

and uh I should do that. How do they

know that the state of the cluster is

not matching the state of the you know

uh desired uh versus current state? How

do they know about it? So let's get uh

let's let's learn that now. So if you

want to know how to build an operator,

the best thing to use is an already

available framework which is called cube

builder.

There are also some other frameworks

that helps you to build cubernetes

operators like operator SDK. However, um

cube builder is also one of the very

famous operator frameworks that allows

you to write your own controllers for

kubernetes. This is for people who are

using Kubernetes and they want to

develop a very indepth uh knowledge of

how Kubernetes reacts on certain

resources, how the operator loop

functions, how is it identities,

how would you know um you actually

compare the state to the desired state.

What is a web hook? How does it work?

How do you implement versioning with a

cubernetes operator?

That's all which is very very inbuilt

and which is very simple with cube

builder. So this allows you to have a

starting point without spending so much

time on what is going to be my project

structure. How would I you know uh

structure my code? How would I structure

my test cases? Um how do I generate my

um metrics? How do I add a locking into

my soft into my controllers? Am I going

to have a leader election? How do I

implement a leader election? How do I

expose a metrics? on what port do I

export the metrics? All of that is taken

care by your builder. What it does is is

it allows you to have a directory

structure in which it has the

boilerplate code for building your

Kubernetes operators already there

thousands of lines. Uh instead of you to

have to write it allows you to focus on

the business logic. It allows you to

focus on what is going to be your

specification of the custom resources.

It allows you to tell what to do in

order to you know how to react in case

there is a change in those custom

resources. That's what it allows you to

do instead of uh looking at how do I

start with an operator in the in the

first place. It also lets you generate

the role based access control. It lets

you generate the cube um um what's it

called? It lets you generate the the

customize resources as well in case you

want to deploy your operator into

different places. It also lets you wrap

your operator into a Helm chart for its

own deployment. So that um it can be

used in any cluster regardless of

whether you are running on cloud,

whether you are running on prem on

wherever you are running. It allows you

to version control your APIs as well. So

for us, let's get started with that. And

the first thing you can do is you can

quickly install

um install cube builder. Let's go there

and installation and setup or maybe I

look on GitHub and there should be some

releases um that you can you can

download. Um we can also install uh

using um the installation book. There

are many different ways of installing

it. Either you can download it from the

releases which uh which one uh is

working for you. I'm using a Mac. So I

have got an ARM 64 because I'm using a

Mac and that's my architecture. And once

it's downloaded, I think you can also

use Buu. I'm not sure if you can but um

how can I install that but as I show you

there you go. So you can install Cube

Builder using a very simple third

command.

Now first thing that Cube Builder needs

or what you do with Cube Builder is you

create a project. Now a project, think

of that project as a collection of your

APIs that you will be building and it's

a simple directory structure that allows

you to initialize um you know um your

your APIs and let's do that now. So

first thing we will do I have cube

builder cube builder version already uh

which is which is available 4.5.1. I

think the latest one is 4.7.1.

I'm not too far behind but that's okay.

So I've got the cube builder and the

first thing we will be doing is we will

create a project where we will be

hosting or we will be you know um

building our API. The first thing cube

builder uh in it and here is the

important thing when you are building

your custom operator um let's say you

are working in a company called uh

example um you want to build your uh

custom resources in a certain domain

which makes it easy for Kubernetes to

know where this operator is coming from.

If you do cubectl

API resources and if I do less here you

can see every resource in Kubernetes is

actually its own identifiable API um

every resource that we see is an

identifiable API c um API resource for

example if I uh look at let's say um

AI services here for example hub.tra.io

io/me1pha

1. We will talk about what the group

version kind is. But uh just to just for

you to know uh you can define the domain

in which your API should be uh declared

in which your API should be built. So

for example, I could say uh Q builder uh

in it I want to be building things

related to cloud and let's say I work

with um um Netflix for example and my

products should be under the domain of

netflix.com in this case I'm using

cloud.com and the repository in which my

um in which my code would be hosted just

as a project descript encryption. What

it does is it writes the customized

manifests for you. So you can have it

deployed in different clusters based on

your requirements. It writes a lot of

scaffolding code for you. And what it

does is is it creates you a directory

structure. It writes you a docker file

which you can use to build your operator

into a uh into a deployable image. It

creates you a make file that uh you can

use to generate your custom resource

definitions. Maybe I open this in VS

Code. That would make more sense.

Um maybe I open this here in cursor.

That would make more sense. So it gives

you a make file that lets you generate

your um your you know uh your RPA lets

generate your custom resources, custom

resource definitions. It helps you

deploy those into a cluster and install

them from the cluster. If you are doing

a local testing, this make file is

really really um helpful. And this is

where is going to be your project. Uh

this is the project uh information on

where uh what is the name of the

project? What is the domain under which

your project uh is is defined and um and

what is the version of uh of of the cube

builder project that you are using.

Apart from that and this was the docker

file that we were talking about. Apart

from that it gives you this cmd

directory. Now it has already created a

lot of files and a lot of folders for

you. So let's quickly go through that.

The first cmd main.go is actually the

entry point of your operator of your

controller. So this already is done for

you. you would have to worry about what

libraries in Go I want to import in case

I want to build a custom operator.

Whenever I say operator um when I'm talk

I'm talking about controller because

that is a loop that actually uh does a

job for us. So you would be thinking

what library am I um supposed to be uh

you know importing for example take this

the client go and the uh client o

package. So this O package is actually

the one that allows you to talk to um

Kubernetes.

It it imports all the Kubernetes client

O plugins in case you were using GCP,

Azure, uh you want to talk to the

clusters. It lets you get the cube

config and this is the package that lets

you work with. You also have a package

for uh importing the Kubernetes API

machinery. We will talk about API

machinery in a in a bit. uh this lets

you define uh do the runtimes that are

needed to define a cubidity schema. How

do you declare a health endpoint? How do

you do logging for your operator? It let

you create a lot of codebase and this is

the main go which is the main file from

which you declare your um your code.

This is the entry point for your code.

We will talk about that when we um when

we write it. You also have a lot of

config folders where you define u how

are you going to uh be working with your

it has some defaults for kubernetes like

your services like your customized

files. It has customization that lets

you deploy your operator to different uh

clusters and name spaces. It has the

customization for your manager which

lets you create a deployment and the

name space in which you want it to be

deployed. It's a fairly straightforward

customization file. It lets you also

create role bases uh access control. It

lets you create cluster roles, cluster

role bindings. Um so it easier for you

to be running your operators. Otherwise,

if you are managing, let's say you write

an operator

which listens on a resource called um

EC2 instance, but it doesn't have the

permission to uh to to be uh you know um

listing EC2 instance in a in a

namespace. You will not be able to

manage those resources in that name

space. So without you worrying about how

does my role based access control would

look like it lets you create a lot of um

boilerplate code along with it lets you

create the rolebased access control as

well uh for you it also gives you end

toend testing so you don't have to write

your own testing fees it lets you help

uh it helps you with that as well and

the one thing that is uh interesting

with that which I was looking for is the

where did that So where is my cmd config

hack? I simply I'm missing Oh yes

because yeah so this is just the project

resource. This is just the project uh as

a boiler plate that cube builder allows

you to do. The second thing we can do

with cube builder. The next thing we can

do with cube builder is to actually

create an API. And this part is amazing.

This is going to be our resource that we

this is going to be our custom resource

that we will be creating. So what we

have just done is what you have just

done now is we declared a project called

cloud.com.

Now with cloud you have many resources

to manage. You might have uh things like

compute to manage. You might have things

like um storage to manage. You might

have things like network to manage

things in compute. Could be uh your um

let's say um EC2 instances you know it

could be your AMI in images for example

they could be your security groups as

well. In storage it could be a EBS uh

EBS module. It could be an S3 bucket

that you want to manage. Uh in network,

you might want to manage a VPC. You want

to uh manage a firewall rule perhaps. So

the thing that I'm trying to say is you

can create multiple APIs in a single

project in a single domain and this is

what we are going to be doing. we will

be building our own API which is going

to be in the compute subdomain and it's

going to be our EC2 resource. So that is

what cube builder allows us to do uh is

to create our own little API. So let's

do that. I would do cube builder create

uh here we go cube builder create API.

The group is going to be compute uh and

kind is going to be EC2 instance.

I want to create the resource. Yes. So

this has created the custom resource and

the custom resource definitions for me.

Uh it has written them on the disk. And

yes, I want you to create the controller

as well. So it downloads um many

different go uh go packages. It also

creates a directory called API/v1.

And this is absolutely

uh important. This is the API the

version of our API and we are building a

file uh we we building a resource called

EC2 types and that is where we define

our EC2 types.code.

Um now once we talk about um now once we

talk about the uh the EC2 type.go we can

take a look at that how does it look

like and this is where the actual

business logic would go for us.

This is where the actual specification

of our API would look like. Now before

you can build your own Kubernetes

cluster, I'm sorry, before you can build

your own operator for EC2, let's let's

see what would this actually look like.

You know how you going to use the YAML

for that? So if I give uh EC2 operatory,

I would probably say um kind is an EC2

operator.

Um meta, it would have some metadata. I

would give it a name and name would be

um my instance and then uh name space

would look like uh default

um

API version. It's defined in

compute.domain.com.

Um, this is a version one of our EC2

operator API. And then I would have two

things. So, every resource you have

would have a spec or almost all of them.

Uh, and then they would have a status

field. And this is something which is

very very important. When you are

writing a custom resource, you have to

define what the resource is going to

look like. What is going to be things in

the spec of your resource and what is

going to be in the status of your

resource. And this is what um the the

file in API v1 EC2 instances.go helps us

to do this. It lets us declare our given

um spec for the resource that we are

trying. Um, for example, my spec would

have um um AMI ID and this is going to

be the my dummy AMI ID and I would have

a key or I would have an SSH key.

This is going to be my key pair that I

want to use on Amazon. Uh I would have a

instance uh let's say I would have a

type. So maybe T3 micro I want to have.

And then you could have a storage and

you would have uh in storage you would

then say um I want a standard disc.

Maybe you could say I want a a

persistence or you could say fast disk

which translates to one of the faster

block devices in Kubernetes because you

want see you all you want to do is you

make you're making the developer life

easy. you're abstracting the actual

details um from the developers. So they

can say okay I could go for a standard

disk of size maybe 10 gigs and fast

would be of size of 50 gigs

that is that is the data that I need and

this would be one of the minimum things

you can use for your cubernetes cluster

and with this spec that you're giving

every resource has a spec and that is

defined for kubernetes it is defined as

at a strruct in collab. So if I uh look

at this DC2 operator, I let's say we

just keep this simple. We're going to

keep these three AMI ID, SSH key, and

type. Um this is going to be my things

that I want to use and all. Let me just

copy the um let me just comment this

out. Where did that go? There we go. So

I define the spec. Now this is the spec

for my uh Kubernetes uh for my operator

and I'm going to say my EC2 instance

spec will contain an AMI ID. It will

contain the SSH key and also the type of

the it will contain the type uh of the

instance that I want to be using. Now

this is where uh it's very important for

you to give these JSON tags because when

you give a request to Kubernetes about a

kind of EC2 instance it needs to

marshall your request. It needs to

understand what is this key uh and what

to do with that is this key is AMI ID

this key is SSH key this key is type. So

these uh JSON tags are absolutely

required for serialization so that

Kubernetes can know this field relates

to a certain um required um key for

example.

Then you can also have the status for

your EC2 instance. Maybe you want to

give out uh things like in in this one

you might want to give uh the space as

probably it's running if your EC2

instance is running or not. Maybe you

want to give out things like um public

IP and that's going to be a 1.23.

And this is what you will be putting in

the status field. So I would say um if I

look in here you see to operator I want

to have phase

um I want to have phase which is going

to be a string this is the type of

string and I want to have uh let's say I

want to have the instance ID as well and

I can just simply go for a public IP. So

these three things are which I want to

um be be having. Now this is very

important when you are using when you

are building resources like this an AI

editor would really help you uh like you

can see I'm using cursor uh this really

helps you to speed up your development

again you are the one who's doing the

thinking you are the one who is coming

up with the spec you are the one who is

coming up with um you know what what

should you be showing in the status

however it helps you as a as a very good

helper

Now you got the spec, you got the status

because these two things are absolutely

important to be um to be in a resource.

Now how would your overall resource look

like? The instance the EC2 instance

would have um the type metadata and

object metadata. So when you see any

Kubernetes resource this kind and API

version this is actually coming from the

type meta. So this meta v1 is actually

you can see this is a package in

kubernetes. This defines the metadata of

any kubernetes resource. This go package

defines the metadata of any uh resource

and has two type of uh you know it has

two strcts there. So the kind and API

version that we see on all the

cubernetes resources it is actually

defined in a strct in Kubernetes called

type meta. And this is what the EC2

instance would look like. It would have

some type meta. So you can see here on

if I copy this probably this would make

more sense.

Let me just copy that all the way here.

Uh and this would be

there you go. So let's comment that out.

Now this is a type of EC2 instance which

is the kind of a EC2 instance. So I got

that. There we go. So the first thing

this kind has is the API version and the

kind. The first thing the resource has

is the API version and the kind. And

these two things are defined by the type

meta. And then we have the metadata of

the object itself and that's defined by

the object meta which contains the name

of the object which contains generated

name of the object the name space the

UID the resource version the creation

timestamp every every object would have

these two um you know struts declared

inside of that which defines what object

it is and second which defines what is

the object's metadata and then You have

the spec where you have defined this

spec and then you have the status which

defines the status of the resource and

this is how an API is created. This is

how you declare what resources are going

to be in your API. Now I don't have to

tell my developers that guys you need to

raise me a ticket so I can create you a

resource in Amazon. Oh, you wanted 10

gigs. I probably gave you 15 gigs. Maybe

I did not hear that correctly. Let me

delete and recreate that or resize it.

You do not have to do that. If I just

give this to my developer,

it is so much easier for them. Maybe I

can have them a simple UI that lets them

declare the name of the instance, the,

you know, the count of the instance,

what storage they want. It automatically

creates me this manifest. And because I

already have a Kubernetes operator and a

you know a controller listening on top

of that, it is very easy for me to track

every request that a developer is making

for these uh instances because um they

are all they can be put into a version

control system. They can be put into

GitHub and you can use our code CD that

makes developers life so easy. They do

not need to know about what is a fast

storage. They don't need to worry about

what is a standard storage. Of course,

they need to know the benchmarking of it

but they don't need to know it is a

persistent disk. They don't need to know

the different type of stoages uh Amazon

has to offer. It is offloading from them

and that is what it makes it very very

simple.

Now things that you see here um these

ones plus Q builder object root true. So

these ones are called cube builder

markers and they are there for code

generation. They are there for custom

resource definition generations for you.

For example, this one says this is

actually a Kubernetes resource. So

somebody could say cubectl get EC2

instance. Somebody could say for example

here is where somebody could say cubecdl

get instance list and that is going to

be uh what is returned this defines it

also has a sub resource called status

which we are defining here above. So

this is what cube builder helps you with

and in the end we are registering our

EC2 instance and EC2 instance list with

the cubernetes schema. this function uh

it uses the resources that we just

created. It gets the APIs that we just

declared and it initi registers that

with the Kubernetes schema which is

actually this function comes from a file

called group version_info.co.

Now this one it's a very simple file. It

uses the Kubernetes schema runtime

package uh from API machinery and the

controller runtime. What these packages

allow you to do is they let you declare

your uh they let you declare your APIs

and the kind to Kubernetes and here you

are saying that you have a group

version. So you're declaring a schema

group version. The group is called

compute.cloud.com

again. So you could say your domain

uh domain was actually uh cloud.com

and then your group was uh compute

uh and then uh your compute.cloud.com

cloud.com and then your version is v1

and then your kind is e2 instance group

and this is how every resource in

kubernetes think of that as a URL every

object on the web has its own unique

identifiable um URL for example

um think of that as kubernetes every

resource is declared in a group it has a

version and it has a kind. Every

resource does that. Every resource has

it. Pod service. If I do that, maybe I

could do kubectl

explain service. You can see here it

kind is called service. Its version is

v1. If you do not see the group, that's

because it is in the core group of

kubernetes, which is uh which is which

doesn't have a name, but it's called the

core group. So is the same for pod. Uh

if you go ahead um here you can see pod

is v1.

So this is why you now understand when

you write kind we are pod API version v1

you are telling kubernetes that this

yaml that I'm giving you it is a

resource of kind pod which is declared

in this group and I want the version v1

for this resource. Every resource have a

group version kind and this code is

actually adding your declared schema and

it is adding your um declared group into

Kubernetes. So it's loading your

resource YAM your actual custom resource

declaration into Kubernetes. So when you

give it a YAML of EC2 instance it knows

what spec this resource has. What is the

AMI ID? what is going to be um the phase

that is running what is going to be the

uh the public IP that I'm going to be

returning so it knows what is your spec

and status that is what we are doing

here we create a schema builder so that

we can add our own schema and then we

have um this this add to schema um it it

does add the type in your group version

to kubernetes and that's where the magic

actually happens this is where you

declare what is going to in your a in

your resources.

Um once you have that then you can also

uh look into another directory that it

has created for you called the internal

controller and that is where the

reconciliation logic happens. That is

where you get the reconciliation logic

uh of what to do. So this one is about

custom resource but what to do on top of

that custom resource? What do I do with

that? That's given in the controller um

package in the internal /controller

directory and there's a file called your

API named_controller.go.

What this does is it creates its own

package and it then creates your um you

know it creates a reconiler.

In this reconiler strct it is having two

um it imports two uh interfaces. one is

the client which gives you the actual

Kubernetes client that you can use to

talk to Kubernetes clusters and then

there's a there's a schema that we can

then use to convert between the YAML

that you are giving and what Kubernetes

knows about you know what is declared in

Kubernetes um resources

then you have some custom markers for

for rolebased access control and this is

where the actual reconcile dilation loop

happens. This is the one uh this was the

actual logic that makes sure your

cluster state is equal to the desired

state. That's the one that makes sure

your cluster state would be um it reacts

on the cluster state and looks on the

desired state and say this is where your

logic will go. This is the heart of your

controller. This is the heart of your uh

of what you are writing what you want to

do with that and then you return a

result

and an error. Now we will talk about um

these two things as well. I'm just

running you through the code when we

write our own as an example then we will

uh we will look into this. Once you have

the reconciliation logic, it is actually

adding um it's adding uh the controller

with the controller manager. So this

setup with manager, it uses the

controller manager to add your

controller too. I think it makes sense

if we talk about the architecture a

little bit of cube builder and that

would be so much helpful. So if I go to

architecture, this is the one that will

make so much sense. what Cube Builder

allows us to do. Oh, wait a minute.

Okay, so the when you run, let me go

here.

When you run uh maybe a little bit

bigger would help.

Let's say here.

When you run a Kubernetes um controller,

the first thing that it runs is it runs

the main.go

program. If you remember, this is from

the cmd/main.go

which is the file here. It starts with

the cmd uh main.go file. So the main go

file is the one which is responsible

when you build your operator into a

binary. Here's a main function that's

the entry point of of the operator. So

let's take a look at its main file from

the beginning. It's part of the main

package and it does import quite a few

of um inbuilt packages from Golang.

However, for it to really be working as

an operator, there are many more

packages that are imported um and that's

from the Kubernetes itself. So let's

take a look on those packages. The first

one that we see here, this is the O

package. And this lets your operator uh

use the exec entry point plugins or um

you know uh talk to your EKS clusters,

talk to your GKE uh cluster API server

or using the OIBC if in case you're

using for authentication. This one's

responsible for making sure that your

operators can use the cube config or the

exec entry points and they can talk to

your cubitus cluster.

The runtime package from the API

machinery is responsible uh to kind of

you know you understand YAML but

Kubernetes does not understand YAML. It

understands objects which are ghost

trucks you know in example. So this one

defines schema. This one's define

objects that can help you to convert

your YAML into Kubernetes understandable

constructs. Kubernetes understandable

objects. And when you do um cubectl get

pods, the YAML that you get is actually

converted from the pod object in

Kubernetes by using the runtime package.

We also have in the API machinery util

package and uh this would be looking

like it's the same package again but

this one's defined in pkg runtime in the

API machinery and this one's defined in

the util uh as runtime and this one is

more like a utility function that helps

your operator be stable in case there

was a panic which is kind of like a

fatal error that your operator got. So

instead of completely crashing the

process, this lets you log that

particular panic and still uh complet

still continuing with the with the

operator process so it doesn't just

completely crash onto you.

We then have uh the client go package

which is again uh this is the I think

the SDK for go for kubernetes and here

we are calling the schema or scheme

package and this one lets you register

your APIs that you have defined the

custom resources. It also lets you

define the pod services the core

constructs of Kubernetes um with your

operator or rather think of it this way

that it gives your operator the

knowledge of the predefined Kubernetes

resources like pod deployment uh secret

services and also it lets your operator

register the EC2 operator um custom

resource that we are creating.

We also have the controller runtime

package and this one right here is the

secret source which is responsible to

have you or to work with a manager that

can help you with clients caches and the

leader election. This one, this

controller runtime is the one that is

responsible that gives you the tools to

construct the controllers that can

listen on changes on your custom

resources and then uh you know they can

uh handle the caches, they can handle

the clients to talk to the API server uh

and eventually um if in case you want to

have early election or not uh that also

is done by the controller runtime. So if

I want to talk about a little bit of the

architecture of how this um controller

would look like. So we would have the

process which is again started from the

main.go

and this main.go would have a manager.

Again you will see this as coming ahead.

But here's where a manager is the one

that manages two things. one, it has a

client and this is used to communicate

to the cube API server and it also

handles the caches of your requested or

um the the the custom resource that was

updated.

Imagine this, you want to write an

operator that reacts on a change uh to

the EC2 operator object and that's where

the EC2 operator object YAML or the spec

will be stored. We're going to talk

about the cache much more in the in the

future in the video. Not right now. It

doesn't make more sense. However, um for

me to explain uh the manager, it does

have the client which is used to talk to

the API server. Then we have the cache.

And here's where the interesting thing

comes into the picture. This is what we

are writing right now. Or rather this

green bit. This green bit right here is

our user provider logic which is what we

are using in the reconcile function.

This controller is responsible for

reacting on the changes and eventually

running the reconiler which is our logic

that tells what to do if in case the EC2

operator object was changed or you know

um whatever change you made to that this

is where it's going to be um this is the

logic which is going to be uh running.

You can also have in the manager in your

controller you can also have a web hook.

This is kind of like the similar um

validating web hook and mutating web

hooks. If in case you want your operator

to also uh serve those web hooks, it's

possible to do so.

Now we also have couple of um we also

have couple of packages for the

certificate watches. This is the one

which is responsible um when you are

working with uh let me rather draw it.

This will make more sense when you are

using things like C or let's say you

have um uh admission control admission

web hook in your operator you have a

mutating web hook.

Here you have a mutating web hook and

your Kubernetes API server. You register

this web hook with the API server and

this can then talk to this mutating web

hook. The API server will simply ignore

or will not talk to your web hooks. You

know, I'm not going to explain the

mutating web hooks or validating web

hooks because this is not a part of this

course. Um, it's something there are

very good documentations that you can

read about. However, when your API

server talks to any of the web hooks,

whether it is mutating or whether it is

validating, uh it has to have a valid

certificate.

It does not talk over HTTP. You have to

have a valid certificate. And a lot of

times you would be using the cert

manager to issue your certificates to

this uh service your your controller

that is hosting the mutating web hook.

Now if in case the search manager uh

again it's used to uh issue certificates

for your web hook and every 90 days I

think by default it will be rotating

your certificates and in this case if

your certificate has changed maybe you

are storing that certificate into a

secret then it is given into the pod. Um

however if this certificate is changed

you will need to restart your

controller. You will need to restart

your controller pod. So eventually the

new certificate is loaded and the next

HTTP request uses the new certificate

which is renewed by the search manager.

This offers a downtime and to fix this

we have theert watcher um package. This

one creates a watcher for the change

certificates and it reloads them on the

fly without you to have to restart your

controller package. So you don't have

any downtime uh in case you are updating

your certificates in case you updated or

search manager did an update for your

certificates.

We also have the health package what

lets you uh expose the the you know the

livveness probes and the readiness

probes that you can use for your

operator. This exposes the health and

the readiness endpoint probes which you

can use in your deployment when you are

deploying this operator and you can say

uh check at this endpoint every now and

then. Uh it's a similar uh it's a

standard Kubernetes livess and readiness

probe. We also have the zap package

which is mostly used for logging. We

then have filters package in the metrics

package here. And this one let's uh I

think this makes sense for me to first

talk about the metrics here and then we

talk about the filters. See when you are

writing your operator with cube builder

it doesn't just let you focus on the

reconciler. I mean this is what your

business logic is. That's what you are

uh supposed to be writing. However, with

cube builder, your operator

which is running in a pod, it by default

exposes an endpoint called matrix. And

this might be looking sim familiar to

you. Um because this is something which

we use a lot in Prometheus. When you are

writing a Prometheus service monitor or

when you are writing a scrape config,

you give three things to the Prometheus

server. the IP or the service name, you

give the port number of the scrape

config and then you also define uh the

you know the path the scraping path.

This same you can use uh with your

operator cube builder. When you are

building an operator, cube builder

exposes the metrics endpoint and this it

it exposes couple of Prometheus readable

metrics like what is the success rate of

your operator? How many times the

reconciler has executed? How many times

it event it resulted into an error? How

many times it resulted into a success?

So it's not uh it doesn't give you an

idea of how many EC2 instances have you

created but rather this is more on the

metrics of the operator itself and then

if in case you want to maybe you you

have a requirement that my operator can

create EC2 instances but I also want to

know how many it has created

successfully. So you know you can also

expose your metrics you can instrument

your code with Prometheus uh go packages

and as soon as you were able to create a

VM you know uh on on Amazon we'll look

into the code in the future uh in the in

the further parts of the video uh you

can then increment your uh AWS instance

count uh to one because you were able to

create just one more um instance and

then you can expose this to the metric

endpoint. The thing that I'm trying to

explain here is it's already done for

you by cube builder and by default there

is no username or password. It is open

to everyone and then you can use

Prometheus with a scrape config to

scrape this metrics the operator related

metrics uh into Prometheus and show that

onto Grafana.

However um you can also then use this

filters uh package. This lets you define

some sort of authentication that this

metrics endpoint is not publicly. It it

should not be publicly accessible. I

only want to um I I only want to allow

someone who has this username and

password. Uh I want to have some sort of

authentication on this matrix endpoints.

And these are the this filters package

provides us these functions where we can

use um these authentication gate um

gated authentications for our metrics.

We then have the web hook. Again, this

is the package which is responsible for

you to create these validating web

hooks, mutating web hooks. There are

many many videos available. Uh we also

did a live stream on cube simplify of

creating your own validating web hook.

You can definitely take a look at that.

I'll put the link of that in the

description. And uh this one helps you

declare your validating and mutating web

hooks. These are core heart of your

operator. You know without these

packages it without cube builder using

these packages it would be very very

difficult for you to build an operator.

So cube builder is really good in terms

of scaffolding your project. When I say

scaffolding it means it is it gives you

a very good blueprint. It gives you a

lot of boilerplate code which again you

can uh refactor but to begin with you

only focus on your reconciler logic and

that for me it's amazing.

Now here's where the repository where my

code is going to be in the API v1 and

this is where I am uh calling my custom

resource definition which I declared.

You remember we had API then v1 and then

we had the EC2 instance right here. This

was our spec of the EC2 instance. That's

what we are calling in uh in the the

main.go. So I am calling my um my v1

with the name of compute v1 and then I'm

also calling my actual controller logic

which has the reconiler or this is where

my reconiler logic will be in the

future.

Now coming forward we have couple of

variables. This setup log is fairly

simple. This sets up a logger for our um

you know for our controller and the

scheme that you see here. Think of this

as a phone book. This is an

instantiation of the new scheme

function. The scheme is acting as a

phone book. It is acting as a registry

where you will write all of your objects

that you want Kubernetes to know about

or rather your operator to know about.

And that's what we do here in the

function in it. We use the util runtime

which is available here. for this

runtime package.

And here's where we have a must

function. So what this does is in case

this must function returned an error um

in case this must function you know was

not able to register if there was a

panic the program will stop right here

because your operator is completely

useless. um yet your operator is

completely useless if it doesn't know

about the core uh API types like pod,

deployment or rather also your own EC2

instance. So we register the default um

core u we register the default API types

and you can look at that using um

cubectl. Let me increase the font a

little bit. We can do cube ctl uh API

resources here. You see? So think of the

phone book which is our scheme. We are

adding all of this um to our phone book.

So we are telling our operator this is

what we have available uh all of this is

what we have available in our uh AP in

our cubernetes cluster and then we also

add our own default u our own custom

resources which is what we are calling

from the API B1. So essentially we are

telling Kubernetes that the scheme that

we declared over here it's an empty book

and in that empty book using the add to

scheme function which is here given uh

to us by the client go scheme we add the

built-in types so our operator knows

what built-in types are available into

Kubernetes

and also we add our custom type which is

the EC2 instance and then our uh

registry or the scheme is a complete

catalog And that's what our operator

would be able to use. Now,

now here's the main function. This is

where everything starts for any go

program. And we are defining a couple of

variables. For example, I want to define

the metric address um on which IP of my

port the metric would be listening to.

And once we have defined these

variables, we also define some flags

from the command line uh when you are

running your your you know when when you

build this with go build and when you

run this binary you can give these uh

command lines as metrics address probe

address you can define leader election

and all that. So we define the IP

address on which our metrics should be

sobbed. We declare some variables which

is the path for our metric certificates

because um just like web hooks can be

served over a certificate we can declare

that our metrics also is declare is you

know um accessible over HTTP or it needs

a TLS config as well and that's what we

can define with these variables what is

the path of our certificate what is the

name of the certificate and the key we

want to use for our metrics. The same

goes for our web hook.

Now there's a very good fun there's a

very good um concept that operators can

help you with or rather when you are

running distributed systems like HCD or

especially when you talk about your cube

uh controller manager

see that is also a controller what we

are writing it has many it's a

collection of multiple controllers but

this runs as three different pods in

your cluster or rather It runs each one

on the master in your cluster. The thing

is when you are writing a controller uh

it is very important of how these

controllers are running in parallel and

do they all make changes or not. For

instance, uh take if I was running two

copies of my EC2 controller. So this is

one controller and this is another

controller and there was an update um

which lets me create

uh an instance

you know uh I did an update I created an

object of EC2 instance kind there was

update and this update was seen by both

of my uh controllers controller number

one and controller number two

they both are going to go and create me

an EC2 instance and this is not What I

want I do want high availability but it

should be active passive. There should

be one leader. There could be multiple

replicas for high availability but only

one at one time should be running. And

this is what leader election u you know

um uh is something that you can use and

cube builder makes it very easy for you

uh that it allows you to uh declare the

leader election with a simple boolean.

So in that case this is also running.

This is also running but this is a

leader. So if in case an update

statement or an event comes from the API

server only this one is seeing it and

only one instance is created which is

what we want to do. The other one is

there but it's not the leader. If the

leader is no longer running or or

automatically it's going to become the

leader and this will be then serving

your requests for the EC2 instance

custom resource changes. This is what uh

leader election means and then you can

enable if in case you want to have

reader election and you can run your

operator into high availability. We

define the probe address on which your

uh health probes are available. So you

remember this this package which is the

health where you declare your health's

endpoint and the ready endpoint on what

port number uh they are exposed by

default the port number I think is 8081

here which is the health probe bind

address

the command line flag and this is the

variable that is going to be responsible

for it. Do you want to use secure

metrics or not? And this this variable

secure metrics and metricsert paths uh

name and key they are related because

you can say I want my metrics to be

exposed over h over TLS and if you say

that you want them to be over TLS then

you can define your metric certificate

paths the certificate name and the key

otherwise there's no need for that. Uh

you can also say if your operator does

enable HTTP2

or um HTTP you know it does not enable

HTTP2 and then we have a list of

functions uh that are R TLS options.

I'll make it simpler explanation as we

go ahead. So we declare a couple of

variables we declare a couple of command

line flags. We define some options for

our logging that this is development

true. when you say development, it

actually um gives you a stack trace on

warnings as well. Um it doesn't give you

any sampling. Uh if you go for

production, it only gives you a stack

trace on um on errors and it does do a

sampling for you. So if in case you are

deploying this to production, that's

something you should always consider um

development as false.

Now we set up a logger. We uh we passed

all of our flags of the CLI that was

given by the user. We um you know we

define our options for logging. We

create a new logger. Essentially what

we're doing in this line here is we are

setting up a new logger with our zap

options or with our logging options.

Now with your TLS when you have this TLS

config it's kind of like a list of

options that you can do. One option here

is if you want to disable uh you know uh

if you do not enable HTTP2 here in case

you are disabling uh HTTP2 you can

append that to your TLS options. So we

say in this case uh my um you know I did

not enable HTTP2. So for me in the TLS

options it would be I disable the HTTP2

and I only enable the version 1.1 of my

HTTP because I'm disabling the HTTP2.

Now here's where you create some

watchers for your certificates. You

remember we talked about these

certificates for the metrics and there

could be certificate for the web hook

because you can expose both of them um

over over TLS. So the certificate could

be for your web hooks. The certificate

could be for your metrics. And we have a

we have a cert watcher. So essentially

what happens is let's say uh this is

what I already explained. You have a

cert manager. The search manager renews

your certificates on the disk. This

watcher will be detecting those changes

on the certificates. It will load them

into the memory in the current pod in

the current operator. It does not

restart the operator. It does it on its

own. There's no there's no downtime.

There is zero manual intervention.

Otherwise, you'll have to um restart

your your operator because your um you

know your certificate was updated by the

search manager.

We define our TLS options which is again

a list of functions uh that returns us a

TLS config and we um instantiate a new

variable. So it's kind of like we are

creating an alias and this is the one um

by by this time the TLS options is a

default TLS options um that we would be

using and we declare a new variable and

we set that as a value. So we can

customize um the TLS configuration for

our web hook server uh if in case we

want to use a watcher or in case you

want to even use a certificates or not.

So uh it's easier for us to customize.

Now if you really gave a web hook

certificate path which is here if you

did give a certificate path that means

you want your web hook to actually be

serving over TLS and that's the that's

the thing then if the length of your

variable uh is greater than zero we will

say initializing web hook certificate

watcher and I will be then using the TLS

as well and I will be using the

certificate. So we define a variable

error and here's where we create a new

watcher for the certificate uh path and

the certificate key. If in case there

was an error, you just simply exit one

because you wanted a TLS config for your

web hook but you couldn't get one. So it

makes sense to stop right there. And

here we are adding a new option to our

web hook TLS uh options. this variable

it contains right now till this point it

only has one TLS option which is disable

HTTP2 that's what we we did here you

know uh by this time it only starts with

one uh TLS option which is disabling

HTTP uh 2 and if you have given a

variable um if you have given a webbook

certificate path we then append onto

this TLS options that we do want to use

um another we do want to use a web hook

uh certificate and this is the get

certificate function from the TLS config

that gives us the name or the

information of the certificate we want

to use for our web hooks and here's

where we are creating a new web hook

server with these TLS options

similar thing happens when you are

working with a metric server options so

these are the metric server options uh

where we define the bind address on

which our metric is going to be exposed

this bind address is 80081.

Do you want to use secure metrics or

not? And what are the TLS options?

Again, by this time we are just

disabling the you know um the HTTP2.

Uh we don't have any TLS right now

because if you do not do uh if you don't

give secure metrics which is as a

boolean if you do not give um secured

metrics then there would be no TLS

options. you only work with HTTP 1.1,

you disable HTTP uh you know uh you

disable HTTP2 but if in case you did

give secure metrics you will be using

some sort of authentication

um um that your metrics endpoint is not

publicly uh it's reachable but not

accessible. There is some sort of

authentication and authorization and

only the authorized users and service

accounts can access your metrics.

Now u this was the metric service

options that we started with. If in case

you did want secure metrics you give uh

some sort of authentication and then

this is the same logic that we did for

our web hook certificate path that if in

case you do give you know your metric

certificate path you create a watcher

like we did for our web hook. Uh there's

a watcher which is for our metric

certificates. Then we append uh the

metric certificate option TLS options uh

with the certificate uh information.

Essentially what this does is if you did

give me a certificate path if it's not

zero the length of the certificate path

is not zero you give me the path of the

certificate I'm going to run your metric

server with the TLS option that that

serves the certificate information.

That's essentially what it is doing. So

you should not get confused on uh on

what this is happening, what this is

doing. I just told you. If you do give

the parts of your metric certificate,

it's just going to expose your metrics

endpoint on this certificate that you

have given. The same thing happened

here. If in case you did give a

certificate for your web hook, it's

going to expose your validation or

mutating web hook over with this

certificate information.

Uh and here's the one which is quite

interesting. This is the from the

controller manager from the controller

runtime. You see this is the one which I

just showed you. This one uh lets us

create a manager. Within the manager you

can have multiple um controllers. It

looks something like this. So here I

have in my main.go file

um this is my operator. This is the

main.go file. In here I have a manager.

Oh, wait a second.

This is my manager. Let's take it this

way. And within my manager, then I will

have my controller. And I can have

multiple controllers in a manager. If I

wanted to uh write something about this,

this is my controller.

This is my main.go go which is

responsible uh for creating a manager

using the controller runtime and then

the manager is responsible to or it's

our responsibility to register our

controllers

with the manager

and that's essentially what we're doing

here. So once we declared all the

variables, once we gave all the flags,

once we defined all of our TLS options,

once we have configured if we want to

use TLS for our web hooks and metrics

and if in case we want to use

authentication with our metrics or not.

Once all of that is sorted, we start or

we use the new manager function that

returns us a new manager which is

available here. This is our manager.

This variable has our manager with all

these options. What is the scheme? So

our controller knows about all the

resources, custom resources or the the

core resources available in Kubernetes.

What are our metric server options? If

in case with the metric server options,

do you want secure metrics or not? You

know, uh what is the port number for

your metrics that you are binding to?

What is the IP address for the metrics?

What is the endpoint which is usually uh

by default/metrics?

And then if in case you have given some

um certificates

the same thing happens for our web hook

server is it secure in terms of have you

given certificates to that or not. Uh

and that is our web hook server. Um we

then declare the health probe endpoints

which is um which is a probe address

that is um where did that go? 8081. This

is what your livveness probe and the

readiness probes will be looking into

the container when they are doing a

probe.

And here's where the leader election

because when you are creating a manager,

the manager should know uh are you

looking forward to have a leader

election and you should definitely do

this when you are building an operator

that you want to run in multiple

replicas in multiple pods. There should

be only one which is both of them are

running but only one is active at any

time. So this is this is absolutely your

responsibility um that you can enable

the leader election and then if you

could not make the manager because new

manager returns you the manager and also

an error. So if you could not create a

manager or you give the error that I was

unable to start the manager and you

simply exit because if you don't have a

manager, you don't have anything. You

don't have a controller. So that's

that's the over um that's the one that

looks on your controllers. If there's no

manager, there's no reason to continue.

Just just exit right there. And that's

why we use the OS package. Now once your

manager is created we need to register

our controller which is the the custom

resource which is uh what we need to do

here. So if you were able to create the

manager we are using you know um we from

the EC2 instance reconiler we define the

client and we use the manager.get schema

which tells our manager what is the

schema of our EC2 instance. Essentially

we use a function called setup with

manager and this one sets up our EC2

instance custom resource with the

manager and which is available here in

the EC2 instancecontroller.go file. This

is this is the one uh which is where our

reconciling logic is and where our

reconider logic will be. So it sets up

our controller in here in the main.go

go. At this point once we started our

manager, we set up or we add our um you

know we add our um custom resource or we

add our controller to our manager. So

the manager knows that I have this

particular controller. This is what I

need to listen on to if any changes are

done to this custom resources and this

is what the logic is what I need to run

with uh with the operator.

Now it's also interesting here if in

case you were having some certificate

watches if this was not nil you add the

certificates to your um you know you add

the certificate watcher to the manager

for your metric server for your web

hooks again um we don't we don't use