Here's a problem. You're working at

Netflix and you have millions of users

trying to use the service. For example,

a user might be trying to add movies to

their list, find the next recommended

show, view watch history, update their

profile, and manage their subscriptions.

And while all of this is going on, you

also have a system that also needs to

update internal analytics, billing

systems, email notification service, and

update recommendation algorithm. Now

once you scale this from a single user

activity to millions and millions of

users, you start to get a scope of how

complex this entire operation is and one

of the major complexities here is we can

start out by abstracting away all the

dependent components here into two main

categories producers and consumers. For

example, producers are any entity that

produces facts. So in our case it could

be client apps where users perform

actions like start playback, add to

watch list, search or thumbs up and it

could also be any of the microservices

that create events like register device,

process payment, renew subscription. So

in other words, producers play the role

in creating signals and delegate the

responsibility to an external entity

that can actually take them. Consumers

are components within the system that

actually take the signals created by

producers and react to them. So if an

email relay server needs to send out

forgot password email for example or

your push notification needs to show on

devices or running database searches for

movies and TV shows. This type of

abstraction between producers and

consumers is how Apache CFKA tries to

frame the problem. By essentially acting

as a broker, Apache Kafka can really

focus on getting the messaging pipeline

implemented to prov for example popular

applications like Netflix require

extremely low latency because people

rely on Netflix to work very fast in

providing what they actually need in

seconds. So that means when topics need

to be stored in a way that can be

retrieved at a near instantaneous speed,

Apache Cafka is highly asynchronous by

nature and leverages techniques like

zero copy and sequential logbased

structure where you can essentially

optimize how events are stored and

transferred to reduce the overhead of

writing and transmitting data allowing

topics to flow through the system with

extremely low latency. Other areas that

Apache CFKA helps is what's called

batching. As we saw earlier, when

millions and millions of users are

interacting with their system to

generate events, the producers can

constantly open and close network

requests, which could certainly add

network overhead that could be avoided.

In other words, if we batch multiple

records into one single request before

sending them to the broker on a mass

scale, looking at millions of

interactions that occur, this could

drastically reduce the network load

that's required to handle this highly

responsive and reactive network of

systems. And this type of batching

occurs not only on the producer side but

also in how each topic is partitioned.

Essentially, you can have the topic

contain multiple partitions that all

have their own different consumers that

are designated to process specific

partitions but all in sequence. This

type of partitioning allows true

parallelism and scalability where you

can route one single topic in various

areas within the