In this video, I'll show you how to

build a production-grade AI web agent

that has access to web data and could

scale to millions of users. We'll do

this by using Python, Ingest, Bright

Data, and OpenAI, although you could

replace a lot of these tools with other

ones. And the point of this video is

going to be to focus on scale. A lot of

people build simple AI agents, but they

fall apart when even a few hundred

people try to use them. Here, I'm going

to show you the real tools that allow

agents to actually scale to hundreds of

thousands, if not millions of users and

requests and handle things like caching,

retries, fallback, scalable web

scraping, and all of those things that

never really get talked about on

YouTube. This video is not designed for

complete beginners. You should

understand how to code in Python

already, but all of the agentic features

I of course will go over and you'll see

how to build something that scales,

which is the whole point of this video.

So, with that said, let's get into it.

Let me give you a quick demo of the

finished product and then we'll start

coding it out. So, I'm on the computer

and I'm going to give you a quick demo

of the finished product. Now, what we're

going to build here is a newsletter

generator that could scale to millions

of people and make a lot of money. Now,

the main focus is not going to be on

this specific idea and kind of the exact

project that we build, but more so on

how we can make this scalable so we can

apply to any AI project that we build.

Now, on my screen, you see what's

referred to as ingest. This is an

orchestration tool that makes it very

easy for us to work with AI agents and

gives us a lot of really advanced

features that allows us to be capable of

this scale. Now, you can see that I have

kind of a function call to my AI backend

here where I wanted to generate a

newsletter. And when that call comes in,

we get all of this logging information

automatically handled for us by ingest

as well as we can go and view every

single task that occurred in our kind of

run right in our function run. So we can

see the run that was was invoked with

this. So we had a topic for our

newsletter of AI agent max articles of

three. We then searched for articles

using bright data which gives us this

ungated web access and can access

sources that are typically hard to get

to like Reddit for example for

up-to-date news. You can see all of the

data returned from Bright Data is right

here. Then we generate the newsletter

using AI. So we actually call out to

OpenAI and we can generate that and see

the response. And then we're able to

save the newsletter which is right here

and it specifies the file path. And then

we have the finalization. And we can see

the amount of time it took for every

single step. Now, if I just quickly show

you a demo of the newsletter, you can

see this is what the code looks like. We

have this newsletter folder. And if I

open it up, let's get the markdown

preview. You can see a quick example of

the newsletter that was generated. And

of course, we can make this much more

complicated if we want to. The point is

we can actually generate these at scale.

So, what I just showed you is that we

had kind of one newsletter we could

generate, but I want to show you what

happens if I try to generate 100 at the

same time. So, I'm going to run this

script that I've written, which

essentially generates 100 newsletters or

at least requests to generate 100 at the

same time. So, when I do that, you can

see there's all these different topics

that have been provided. And if I come

here to ingest, now we can see all of

the runs in parallel that are showing

up. Now, this is the really interesting

thing about ingest and how we're going

to write this code is that for any given

run, I can click into it. I can see

exactly what's happening in real time.

And we can see that by using bright data

here, the articles were searched

extremely fast. You can see all of them

ran in parallel and kind of worked at

scale. And then we're typically just

waiting on the generation from our LLM.

So, that's the longest running task to

actually get the result. But all of this

is happening in real time at scale. And

again, you can see all of these

different results here popping up. Some

of them are still running, but generally

we're able to handle any number of

requests based on the way that we have

orchestrated and written this code,

which I think is really interesting. So,

with that in mind, what I'm going to do

now is kind of clear all of this, go

into a code editor. We're going to start

setting all of this up, just so you have

an idea of what we're going to do. We're

going to get all of our keys, all of our

access, all of the tools that we need.

Then, we're going to start setting up

the ingest server. Then we're going to

deal with write data to collect all of

the web data that we'll use to generate

the newsletters because what we'll do is

we'll search sources like Google as well

as Reddit. We can get data from that and

we can use that to give us upto-date

news effectively for the newsletter

which is kind of the whole point. So

anyways, let me clear this and then

let's start writing some code. So we're

going to get into it here, but I just

quickly want to touch on the key tools

that we're going to use in this video so

you can understand why we're able to

actually make this agent scale. Now,

first, as I mentioned, we're going to

use Inest. This is a platform that

allows you to scale the agent and

essentially the orchestration and wrap

all of the different tasks that are

going to be performed into kind of

reproducible individual steps. We'll

talk more about this later, but you'll

see that we can do things like infer the

AI, run a particular step, and that's

what gave us that nice dashboard where

we were able to kind of see into what

was going on with our AI agent. Now, in

terms of actually producing the content

that our agent needs to generate the

newsletter, that's where we're going to

be using Bright Data. Now, Bright Data

has been a long-term partner of this

channel, and they have sponsored this

video. They are free to use and to

experiment with, like I'm going to show

you in this video, and they provide kind

of unfiltered, ungated access to the

web, which allows you to collect all of

the data that's required for AI agents

today. This means you can get real-time

data from sources that are typically

very difficult to scrape and to collect.

And Bright Data has various different

tools you can use. For example, you can

extract, search, crawl, navigate. You

can do data pipelines, go to web

archives. In our case, we're going to be

focusing on the searching because we

want upto-date information from a search

engine like Google for example or like

Reddit. But there's all kinds of

different services essentially that they

provide. So that's what we're going to

be using. And the reason why we use

bright data is because it's built to

scale. You can send an infinite number

of requests to it and it will

automatically scale up and return them

to you in parallel which allows us to

build this agent because we don't have

that limiting factor of the web search

which you typically would have if you

were going to build your own custom web

scraper for example or use a lot of

other services out there that aren't

built to scale. Okay. So what we're

going to do is go into PyCharm. This is

where I'm going to start writing my

code. And you'll see that what I've done

is I've just opened up a new folder

called Newsletter Agent. Now, I prefer

to use PyCharm in large Python projects

because it is designed specifically for

Python. I also have a long-term

partnership with them. So, you can check

out PyCharm for free from the link in

the description. Like I said, you can

use it for free and check out their pro

subscription which has a lot of advanced

tools and you'll see in this video why,

you know, I prefer to go with an editor

like this when working in Python. Okay,

so from here, what we're going to do is

we're just going to open up the

terminal. From the terminal, what I'm

going to do inside of this new folder is

I'm going to init a new UV project. So,

I'm going to say uvaninit.

That's going to create a few files for

us like main.py pi project.toml. Now,

inside of here, what we're going to do

is we're going to start adding the

various dependencies that we need for

this project. So, we're going to type uv

and then fast api

ingest with two ends. Python-env.

We're going to use this to load in

environment variables. Uicorn and then

lang chain

dash and then bright data. They now have

a lang chain package which makes it very

easy to interface with this tool. So

we're going to go ahead and press enter.

That's going to install all of the

packages that we need. Perfect. Now that

we have that, what we're going to do is

we're going to start setting up some of

the environment variables that we need

for this project. So we're going to make

a new file here. This file is going to

be called env. Inside of env we're going

to create two variables. Let me just

close these pop-ups here. The first is

going to be open AI API_key.

We're going to need this to have an open

AAI API key in order to interface with

an AI. Then we're going to have the

bright data

API_key.

And this is going to be our bright data

API key so we can use those web access

features. So let's start gathering these

keys. The first key that we'll get will

be from OpenAI. In order to get that,

we're going to go to

platform.opai.com/api.

openai.com/api-keys.

You do need to have an account on here

and it will cost a very small amount of

money to use the key. Should be

literally like a scent. What I'm going

to do is go create new key. I'm going to

call this web AI

agent and then create the key and I'm

going to copy it and not leak it to you

and place it inside of my environment

variable file and then save. So now that

we have that, I'm going to go and

collect the bright data API key. So, in

order to collect the Bright Data API

key, you will need to create an account

on Bright Data. I will leave a link

below where you can do that for free and

get some free credits. And what we'll

need to do to get the API key is create

a new search zone. Once you create an

account here, you very likely will be

brought to a page that looks like this.

What we'll want to do is go to proxies

and scraping. And from here, we're going

to press on add. And then we're going to

add a SER API zone. So with bright data

you can access all of these different

proxies like residential proxies, data

center, etc. You know, mobile proxies.

You can use their unlocker API which

will automatically solve captures and

remove things like IP bands and um rate

limiting that you would get if you do

traditional kind of web scraping. You

can use their browser API, same thing

for doing kind of remote browser uh web

scraping. So you're not doing it on your

own machine. or you can use the SER API

which is the main one that we want to

use right now which is going to allow

you to scrape search engines. So in this

case we're going to focus on Google. So

we're going to press SER API. For the

name I'm just going to call this SER API

and we will use this name later. So make

sure you name it exactly this. We will

turn on the capture solver so it will

automatically solve captures if those

are presented to us. And I'm going to go

with standard for Google ads but you can

go maximize if you're interested in

grabbing the ads that appear. There's

also some advanced settings. We don't

really need to look at those right now.

I am simply going to go and press on

add. And then I'm going to go to yes to

create the new zone. Once the new zone

is created, then I will show you how to

get the API key. Okay. So, in order to

access the API key, it should show it

directly here inside of the zone. So,

you can just copy the key and then take

the key and put it right inside of your

environment variable file, which I just

did right here. Okay. Okay, so now that

we have the keys, what we can do is

start setting up some of the Python

files we're going to need for this

project. So we're going to have this

main.py file. However, I will just clear

everything that's inside of it. And let

me quickly configure the interpreter

here in PyCharm. So we no longer get

this message. Okay, so the interpreter

is configured as we can see down here. I

just selected the one for this project.

What I'll do now is make the other files

that I'm going to need. So I'm going to

make a new file and I'm going to call

this the newsletter

service.

Let's type this correctly. py. I am just

going to cancel that. I don't need to

add that right now. Then I'm going to go

new. I'm going to make another new file

and I'm going to call this custom

types. py for, you guessed it, some

custom types that we'll need for this

project. I'm then going to make another

new file. For this one, I'm going to

call this the prompts.

py, which is going to include the

prompts for our AI model. Okay. And that

is about it. We just need one more. in

this last one is going to be called run.

py. Okay, so now we have the Python

files that we need. I believe we have

five of them. And we can start writing

some code. So what we're going to do in

order to get this up and running is

we're first going to create our own web

server that's going to run the ingest

kind of code. I know this is going to

seem a bit weird until you actually see

it in action. Essentially, we're going

to have our own server that's going to

be served using fast API, which is going

to allow us to have an endpoint that we

can hit that will start generating a

newsletter. Now, once we create this

server, we're going to run it. So, we'll

just have a normal kind of API running

in the background. Then, we're going to

run the ingest CLI or the ingest dev

server. Now, what that's going to do is

that's going to create that dashboard

that you saw, which is going to handle

the flow of requests to our server. So

effectively what's going to happen is

we're going to send a request to the

ingest dev server. The ingest dev server

is then going to take that request and

proxy it to our backend. So where we're

actually running the AI agent and it

will automatically handle things like

retries, caching, tracking how long

particular tasks take to run, doing all

of the asynchronous kind of scalable

stuff for us. So there's kind of these

two com uh components that are

communicating with each other. We have

the ingest dev server which is open

source free. You can run it on your own

computer. And then we have our server.

And our server is what kind of has the

custom code and is generating the

newsletter. Then we have a client. And

the client is just simply the person or

the application that actually wants to

request the newsletter to be generated.

And I believe if we go over to ingest

here, we can see a quick kind of diagram

of how this works. So here you go. You

can see a quick example of kind of how

this works. Essentially, we have some

kind of request, right? This request

goes to the ingest dev server. The dev

server then sends it to our own

endpoint, right? Our application which

then sends it to various ingest

functions which are the parts of the

application which are reproducible and

scalable as you'll see later on. So

that's kind of how the dev server

operates. If we scroll down through

here, there's a lot more information.

You can read more about it. In fact, I

have an entire video on Injest which I

will leave on screen now and linked in

the description in case you want a lot

more detail on it. But it's very useful

effectively. So, I would highly suggest

using it as I'm going to in this video

and just following along and you'll get

the hang of kind of how it's set up. And

then if you want to do more advanced

stuff with it, again, refer to the

documentation or the other video that I

put on screen. And I'm going to close

that for now. Okay. So, what we're going

to do is go back into our code and

inside of main.py, we're going to start

with our imports. So, we're going to say

import logging. We're going to import

UUID, which is a unique identifier,

which we need for ingest. We're going to

import date time. We're going to say

from pathlib import import path. We're

going to say from fast API import fast

API with capital like that. And we're

going to import the ingest.fast_appi

package. We're then going to say

from.env

import load_.env

like so. We're then going to call the

load.env function. And what this is

going to do is load the values inside of

our environment variable file. Okay.

Okay. Now, after this, what we're going

to do is just set up some basic logging

so that we can see results in the

console. So, we're going to say

logging.basic

config. And here we're going to say the

level is equal to logging.info.

This is going to let us see any

information statements in the console.

We're then going to create a logger and

we're going to say the logger is equal

to logging.get

logger and the name of the logger is

going to be underscore_ame

which is the name of this Python module.

We then are going to create the ingest

client. Okay. So to make the ingest

client, we say ingest client is equal to

ingest dot ingest. And let's just make

sure that we import ingest as well. So

importing inest like that. Now from here

we are going to put a set of parenthesis

and we're going to put the app ID equal

to and then we just need to name this

something reasonable. So we're going to

go newsletter

like that. We're then going to say is

production is equal to false because

we're not currently running this in

production. And we're going to say the

serializer is the ingest.pidantic

serializer with two parenthesis like so.

Now the serializer is what's going to

essentially convert our data to and from

JSON as it goes between our various

ingest functions. Don't worry too much

about it, but if you're familiar with

Pyantic, which is a typing system in

Python, that's the one that we're going

to use here. Perfect. Now, what we're

also going to do is we're going to say

path and we're just going to say

newsletters like that. We're going to

say mkdur and we're going to say exists

okay equals true. Now, what this is

going to do and sorry, this is just

exist not exist is it's going to check

if a newsletter directory currently

exists inside of our project. If it

doesn't, it's going to make a new one so

we can store our newsletters there. If

it does, then it's not going to make

anything. Okay. So now what we're going

to do is we're going to create an ingest

function. An ing in ingest function is

something that we can trigger to run.

And within that function we can have

various different steps um that we run

to kind of complete the function or

complete the task. So what we're going

to do is say inest_client

and here we're going to say

function. We're going to put a set of

parenthesis. And for the function we

need to give the function an ID. So

we're going to say function ID is equal

to and then this can just be a normal

string. It doesn't need to be um

something that's like a variable. So we

can say it generates newsletter as the

ID of the function. This is kind of a

human readable name that we'll be able

to see inside of the ingest console. And

then we're going to have a trigger. Now

the way that ingest works is that from

some kind of client. So the client is

like a user for example, you can create

an event. Now these functions that we're

going to define here can be triggered

when a particular event occurs. So in

our case if we generate a newsletter

that's the event we want this function

to run and we could have other functions

that run when that particular event is

generated. So you can have all these

different functions and they can be

triggered with different events

essentially and multiple functions could

run for the same event. In our case,

we're setting up something very very

simple where we just have one injest

function. But in more complex

applications, you have these events.

These events can be triggered by client

or even an ingest function itself. And

if that event is triggered, then this

function will run. So that's kind of how

it works. Rather than directly calling

the function, we kind of emit an event.

This function then goes, oh, okay, this

event occurred. All right, I'm going to

run now because I saw this event. So

we're going to say ingest dot trigger

event. And then the event is going to be

event equal to and then we're just going

to name this newsletter slashgenerate.

Okay. And that's the ingest event that

we're talking about. Perfect. Now

beneath this we're going to say async

define we're going to put a function.

This is going to be um generate

newsletter and we're going to say ctx

which is context is going to be equal to

the ingest dot context.

Perfect. So then inside of here, we'll

just put a quick dock string and we're

just going to say generate AI newsletter

from some topic. And then just to make

sure that this is running because we're

going to do some debugging. We're just

going to log something. So we're going

to say logger.info.

We're going to put an fstring and we're

going to say generating a newsletter.

And then what we're going to do is embed

inside of here the ctx.event.

ID. Again, I know this seems vague

because you haven't used ingest before.

Trust me, it's very simple once you get

the hang of it and we start setting up

kind of a larger structure to our code.

All right. Now, inside of this function,

we're going to have two inner functions.

The first function is going to start

with an underscore, and we're going to

call this search articles. And this is

going to be the function that is

responsible for using bright data to

actually grab all of the articles and

news that we're going to use in our

newsletter. So, that's the first one.

The second one is going to be define

save newsletter.

And what this is going to do is this is

going to, you guessed it, save a

newsletter. So it's going to kind of

save a new file for us. And these are

kind of two steps that we're going to

run inside of this particular function.

Now before we go any further, I'm going

to go into this custom types file, and

I'm just going to define a custom type

that we're going to need to use here

inside of uh inest. So we're going to

say import pi dantic and let's spell pi

dantic correctly. Then what we're going

to do is create a class. We're going to

say class newsletter

request. And this is going to be the

information that we need when we want to

create a new newsletter. This is going

to inherit from paidantic.base

model. And what we're going to do here

is say the topic is string and the max

articles is an int which is equal to

three.

All right. So, what we're referring to

is, okay, we want some topic for the

newsletter and we need the maximum

number of articles that we should go and

search for. By default, we'll make it

three, but you can make it any amount

that you want. Now, what I'm going to do

is just import those types. So, I'm

going to say from custom types, import

the newsletter request. And we're going

to start using this now inside of this

inest function so that we can kind of

type the different data that we have

correctly. So, first things first, we're

going to say our request is equal to and

we're going to say newsletter request.

And then what we're going to do is put

asterisk asterisk ctx.event.data.

What this is going to do is take this

normal JSON data and it's going to

essentially convert it into this

newsletter request pyantic type which is

going to allow us to pass it to the

various functions here as needed. So

what we're going to do now is we're

going to go to our first function here

and we're going to say request and then

this is going to be newsletter request.

So now we know okay if you call this

function we're calling it with this

newsletter request data. So we can

process that inside of the function. For

the next function, we're going to have

request as well. And this is going to be

the newsletter request. And then after

this, we're going to have the newsletter

text, which is going to come from our

model. And we're going to have the

articles like this count, which is going

to be an int. Now, the reason we're

doing this is the number of articles we

request versus the number of articles we

actually retrieve could be different.

So, I want to have those as parameters

in the function so that when we save the

newsletter, let me make this a little

bit larger. Uh, we can know essentially

how many articles we used to create

that. Perfect. Now, what I'm going to do

just while we're here is I'm going to

code out this function and then we're

going to write the logic for actually

generating the newsletter. Um, which is

a little bit more complex. So, in terms

of saving the newsletter, we're going to

say the timestamp is equal to datetime

dot datetime.

Let's spell that correctly. And this is

going to be dot now. We're then going to

say str. So string f time, which is like

string format time. And inside of here,

we're going to put a percent capital y,

which is year, percent m, percent d,

which is day underscore. And then we're

going to put percent, and then capital h

percent, and then capital m, and then

percent, and then s, which is second.

Okay. Then we're going to say the safe

topic is equal to request

dot topics. This will contain the topic

that we want for the newsletter.

And we're going to replace any spaces

with hyphens so that we're actually

sorry, not hyphens, underscores, so

we're able to use this in like a name, a

file name. And we're going to replace

any slashes with underscores as well,

just to make sure that we have a safe

file name. Then we're going to say the

file name is equal to and we're going to

say f and then what we're going to put

is the safe topic underscore the and

then inside of parenthesis again the

timestamp and then we're going to put

underscore and then we're going to put

again inside of braces string and then

uyu ID dotuyu ID4

in parenthesis and then we are going to

at the end of this put square brackets

and then colon 8 and then MD standing

for markdown because that's the file

extension we're going to use. The reason

why we're putting all of this here is I

want to make sure that I have a unique

ID attached to the timestamp as well as

the topic name just in case we have two

topics that actually end up finishing at

the exact same time so we don't get any

error where we accidentally override a

file that already exists. We can save

ourselves from that by just throwing on

a unique ID to the end of our file. I

don't care about having the full crazy

long UID. So I'm just taking the first

eight characters of a unique identifier

that is converted to a string to ensure

that we always have a unique file name

that's safe to use. Hopefully that makes

sense. Then we're going to say the file

path is equal to path and then

newsletter sorry newsletters and then

we're going to put a slash and file name

which is going to combine these paths

together so we know where to save the

file. Then we're just going to do a

really simple file save. So we're going

to say with open file path we're going

to put a w. Then we're going to say as

f. What we're going to do then is write.

So we're going to say f.right. And then

we're just going to write in markdown

format. So to do that we're going to put

an f. We're going to put a pound sign a

space and then we're going to put the

request.topic.

And then we're going to say newsletter

back slashn back slashn. And then we're

going to put the newsletter text here,

which should already be a markdown

format from our AI. Then we're just

going to log something. So we're going

to say logger.info

and we're going to say with an fstring

newsletter saved to and then we're going

to put the file path like so. Then

lastly, we're just going to return some

information here so that we have the

info in ingest. We're going to say the

file_path

is the file path. We're going to say the

topic is the request topic. And we need

to make sure we put this in quotation

marks.

And then we're going to say the articles

found. Again, needs to be in quotation

marks. So articles found is the articles

count. Okay, that's it. That's all that

we need for this save newsletter

function and we'll use that in a minute.

Okay, so let's keep going now through

the rest of the function. So we have our

request, right? So we've got our

newsletter request data. What we're

going to do now is we're going to say

search results is equal to await

ctx.step.run.

What this is going to do is run an

ingest step. Anything that's a step in

inest is going to be retryable meaning

if it fails it will automatically retry

the number of times that we specify. It

also will be traceable meaning we can

see the result of that step. So this is

how injust is kind of set up. You can

have an overall function and then within

that function you can have these

different steps that you run and those

steps give you that observability into

the AI application. So the first step

we're going to call this search-

articles so that we put the name as the

first parameter and then we have to put

the function that we want to call. So in

this case I'm going to say lambda and

then I'm going to put underscore search

articles and I'm going to pass request

as the parameter to this function. Now

what I'm doing is I need to put a

callable function as the second argument

to this step. So because I want to pass

a parameter to search articles of

request which is this right here. I

simply wrap this function call in a

lambda which makes this callable so that

this is the correct format for this

function call essentially might seem a

little bit weird but that's necessary

the way that we're calling this

function. If the function didn't have

any parameters I would just write the

name of the function but because it has

a parameter I put it in a lambda. Now

the next thing we're going to do is

we're going to say newsletter

text is equal to and actually for now

I'm just going to make this an empty

string but we will call a function here

that will generate the newsletter. And

then lastly we're going to say result is

equal to await ctxeptrun.

This step is going to be called save-

newsletter. So effectively we're going

to have three steps. The first is going

to search for all of the data we need

with bright data. The second is going to

generate the newsletter text based on

that data. And the third is going to

save the newsletter using our save

newsletter function. So same thing,

we're going to say lambda and then

underscore save newsletter. And then

we're going to pass our data. So we're

going to pass our request and then our

newsletter text. I don't know why we

have all of this here. And then we're

just going to pass a one as the articles

count, but we could actually adjust this

later on. Uh which we likely will. And

then lastly, we're going to return the

result. Okay, so I know this function is

not completely finished yet because we

need to write some other code, but

effectively what we're going to do is

have these three steps that we're going

to be running. Again, first searching,

second generating the text using AI, and

the third having the results saved to a

file. And then we're going to return all

of that from this function. Now, the

last step here is we need to serve this

function using fast a API and ingest. So

to do that, we're going to say app is

equal to fast API and we're just going

to give this a title. So we're going to

say title is equal to newsletter API.

We then are going to say ingest.fast

API dos serve and we're going to serve

the app the ingest client and then we

need to put inside of a list here the

ingest functions that we have. So we're

going to say generate newsletter like

that so that we serve this function.

Then we're going to put if_ame

is equal to underscore main

we're going to import uicorn and we're

going to say uicorn.run

and then this is going to be app and

then host is equal to 0.0.0.0.

So just run on all available uh IPs

essentially. And then we're going to say

port is equal to 8,000.

Okay. So what this is going to do is

essentially run our app using Unicorn

which is a lightweight web server in

Python. It's going to run our fast API

application which is then connected to

ingest. Ingest is now going to serve

this function and then when we start

running everything which I will show you

in a second, this function will

essentially be ready to go and we could

call it and test it out and make sure

that it works. Okay, so before we go any

further, what we're going to do is just

run all of this code to just make sure

that it actually spins up correctly. We

can't really test it yet, but we can at

least make sure that it doesn't have any

errors. And then we're going to move on

to the next stage, which is collecting

the data and invoking the AI, right? And

that's where it gets more interesting.

So, in order to run this application,

there's two things we need to run. The

first is our server. So, the one we just

wrote, main.py, and the second is the

ingest CLI or the ingest dev server,

which I'll show you in a second. So,

what we're going to do is type uvun and

then main.py. This is going to run our

fast API server. And you can see that's

running on port 8000. Next, we're going

to open a different terminal instance

and we're going to run the following

command, which is npxing

inest-

cli and then dev. Now, when we run this,

it's going to start installing inest.

So, just make sure you type yes to

install it. And then you should see that

it starts running like this. And you'll

see that it's running on port 8288.

What you can do is simply open this. And

you should see that it brings you to a

page that looks like this. And you're

viewing kind of the Injust Dev server,

which it says right here, which is where

we can test everything and make sure

that it's working. Now, from here, what

we're going to do is go to apps. And we

want to make sure that we see an app

registered here. That is our server.

Okay. So, you can see we have local host

port 8000 running. So, we have our

application on local host port 8000,

sorry. And then we have the ingest dev

server. So, both of these are now

running. What we could now do is send a

request to the dev server which will

then forward it over to our server which

would then allow us to generate the

newsletter. Now obviously not all of

that is finished right now but you get

the idea. So that how it would work. So

if I wanted to test this right now what

I can do is I can go to runs I can go to

send test event and then I could just uh

do kind of a mock test event that would

trigger the ingest function. So if we

have a look here at our function the

event is newsletter/generate. So let's

copy that. Okay. So we're going to go

here. Now the data that we're going to

be looking to accept is a topic. So

let's do a topic of like AI agent for

example. And then if I send this event,

you can see that the function is

running. It gives us the run. Now we

have an error here because save

newsletter returned unserialized data

data. That's fine. We can fix that

later. But hopefully you get the idea.

Okay. So essentially it ran. we're able

to see the function run and now it's

going to keep retrying this last step

here because that's how injust is set

up. So I can simply cancel this and then

go back. So that means everything is

working. It's all set up. We're able to

actually trigger the function to run and

we need to write the rest of the code

now. Okay. So what we're going to do now

is we're going to go into our newsletter

service and we're going to start writing

this service in order to actually

generate the data and then generate the

newsletter with AI. So we're going to

start with our imports. We're going to

import OS. We're going to import async

io. We're going to import concurrent

dot futures. We are then going to say

from.env

import load.env. We're going to say from

ingest.experimental

import AI. And we're going to say from

langchain_brite

data import the bright data SER. Okay,

this is one of the various tools we can

use. You can see if I start typing here,

there's a bunch of other ones we can use

as well like web scraper, unlocker, you

know, API, etc. In this case, we're just

using the SER. Okay, we're then going to

call load.env again just to make sure we

load in the environment variables that

we need. Next, what we're going to do is

we're going to create a class. We're

going to say class newsletter

service

From here, we're going to start writing

some methods. So, the first method we're

going to write, we're going to say

define, and this is going to be

underscore search webc_blocking.

We're going to take in self topic, which

is a string, and max

articles, which is an int, and this is

going to return a string. Then, inside

of here, this is where we're actually

going to perform the search with uh

bright data, sorry. But before we do

that, I'm just going to write the other

functions or methods that we need. The

next is going to be async define and

this is going to be search webc

simple.

We're going to take in self

topic which is a string. And then same

thing max articles int and we're going

to return a string. Same thing for now

I'm just going to pass and we'll write

the implementation of this later. The

reason why I need these two methods is

this method will actually perform the

search. This is going to allow me to do

it asynchronously and concurrently so

that I can do multiple searches at the

exact same time without being blocked by

my thread. Here in Python, I'm then

going to say async define generate

newsletter. This is going to take in

self. It's going to take in ctx

topic which is a string and search

results which are a string as well. And

then it's going to return a string which

is going to be the completed newsletter.

Okay. Okay. And then same thing for now

we are going to pass. And now we're

going to start actually implementing

these methods. Okay. So inside of our

search web blocking we're going to start

with a try. First we're just going to

put a print statement so we can see

what's going on. And for the print we're

going to say searching

with bright data

for and then we're going to put what

we're actually searching for. In this

case I'm just going to go comma and I'm

going to put topic. Okay. So we can see

what it is that we're searching. We're

then going to invoke the SER tool. So

we're going to say SER tool is equal to

bright

data SER and we have a lot of options

that we can specify here for the SER

tool to run. So the first is we need the

bright data API key. So we're going to

say bright data API_key

is equal to OS.get

get env

data API_key which we defined inside of

our environment variable file right

here. Okay, so we're going to get that

key. We're then going to specify the

search engine that we want to use here

is going to be equal to Google. We're

then going to say the country is equal

to US. We're going to say the language

is equal to English. We're then going to

say the results is equal to the max

articles and we're going to say parse

results equals true. Now this is an

interesting part about how the SER works

from bright data. It can actually

automatically parse all of the results

that it gets from the normal HTML

response from Google into a JSON format

or in our case a Python dictionary

that's going to be a lot easier for us

to work with. So make sure you enable

this as true so that it automatically

parses the results for you. If you

wanted the raw data not parsed, then you

would make this false. Okay, there's a

lot of other values you can pass here to

use this search engine API. For example,

you can use a different search engine

like Bing, right? You can use a

different country. You can target a

city. You can do all kinds of stuff and

look at all the results and it will give

you results extremely quickly. We're

talking about a few seconds at most and

give you a ton of results that you can

parse through. For example, you can look

at news, you can look at images, you can

look at videos, you can look at just

normal search results. And I'm going to

show you how we filter through that in a

second. So, now that we have the tool,

we're going to make a search result. So,

I'm going to say search query is equal

to and then I'm going to say f. And what

I'm going to do is just take the topic

and I'm going to say news recent

developments. So if they uh search for

like AI agent for example, we're going

to go like AI agent news recent

developments so that we find all of the

news related to that um for our search.

And we could do more advanced searches

obviously, but in this case this is just

the one that I will use. Then we're

going to say the results are equal to

the SER tool and rather than search this

is going to be.invoke. This is as easy

as it is to use this tool. I simply

invoke this and what it's going to do is

scalably go to in this case Google, grab

all of the different results based on

the max articles that we've provided

here and then give it to us in a parsed

JSON format which we can then use and

pass to our agent. So I then I'm going

to do a print statement and I'm going to

say print fing like this and then got

let's do this got results for and then

we're going to put the topic like that.

And now we're going to go and we're just

going to extract the results that we

need. Now this generates a ton of

different results and I don't want all

of the results because if I have all of

the results then that's going to be too

many tokens to pass to my AI agent. So

I'm just going to get the results that

are essentially relevant to me. So I'm

going to say if is instance

and this is going to be results

dictionary and organic.

So these are results that are not

sponsored are in results. What I'm going

to do is say the essential results is

equal to a list and this is going to be

yeah central results. That's good. I'm

going to say for result in results and

then I'm going to look in the organic

portion of the results and I'm going to

go up to just the max articles. So I'm

going to say colon max articles. Then

what I'm going to do is say essential

results.append append and I'm just going

to append a simplified version of the

result because again I don't need

everything. So I'm going to say title is

equal to result.get.

So let's do this properly and then title

and if we don't have that we'll just

have an empty string. Then we're going

to have the description

and the description is going to be the

result.get

and then description and an empty

string. And then we're going to have the

source and the source is going to be the

result.get get

the source and then if we don't have a

source an empty string. Now what I'm

effectively doing is because there's so

many results that Bright Data is going

to return here. I'm just filtering

through them, right? So if you want to

see what the raw response looks like,

you can just print out all of the

results and you can see what's

interesting to you. In my case, I just

care about the organic results. So

that's what I'm doing, right? I'm

looking in the organic results. I'm just

looking at the number of articles I

specified. And for each result that I'm

having a look at, the only thing I'm

grabbing is the title, description, and

source. even though there's a lot more

information associated with it that in

this case I just don't need because I

don't want to overload my LLM with

tokens when I don't need that. So then

what I'm going to do is just tab back

one. So we're outside of the for loop

here and I'm going to return the

following which is going to be a string

and the string is going to be query.

The query is going to be the topic that

was provided and the results are going

to be the essential results. And I'm

just returning a string so I can pass

that directly to my LLM. I then am going

to just have an if statement here. And

I'm simply going to return the string of

results which will just be this if for

some reason this is not a dictionary.

Otherwise, down here, I'm going to have

an except I'm going to say accept

exception as e. And then I'm just going

to return an empty string and I'm going

to print out the exception if for some

reason an error occurred. So what I'm

doing is I'm just putting a simple try

statement, right? Saying, okay, I'm

going to start, you know, searching with

bright data. I set up my SER tool. I

invoke the search. So I get all these

results. I then just simply parse

through the results, get the information

that I need, and return that. And if

there's an error, I just print out the

error and then return nothing because we

didn't get any results. Okay, so now

what we need to do is be able to call

this um what do you call it?

Asynchronously so that we can actually

run this multiple times without being

blocked. I'm not going to explain this

too much in depth cuz it's a little bit

advanced, but I'm going to use async.io

to do this. So I'm going to say loop is

equal to async.io.get

event loop. I'm then going to say with

concurrent and this is going to be dot

futures

thread pool executor as executor. For

the threadpull executor, I'm going to

say the max workers is equal to 10. Um,

you doesn't really matter how many you

put here, but the more you put

essentially, the faster this is going to

execute. What I'm effectively saying

with this is that I want to run 10

threads at the same time. Um, so if we

put more, then it should run more

concurrently, but 10 is kind of fine for

what we need here. And then what I'm

going to do is say result is equal to

await. And then loop.run inexecutor. And

inside of here, I'm going to use the

executor. And I'm going to call the

function self.arch webb blocking. And

then I'm going to pass these two

parameters topic and max articles. Now I

know that this is confusing and a lot of

you probably don't know what this is

doing. Effectively what this is doing is

just ensuring that we are going to be

able to run all of these search

operations effectively at the same time

without stalling or waiting for a search

to return. The reason for this is that

if I don't do this, I'll constantly be

waiting for one of the search results to

come back to me. So let's say a search

takes 5 seconds and I need to do 100

searches. My application's going to take

500 seconds to run when really if every

search takes 5 seconds, I should just

trigger every single search right away

and then I just wait 5 seconds and then

all of the searches are finished and I

can get the results. That's effectively

what this loop is going to do for us

right here. allow us to run this

concurrently using the thread pool

executor which again is a bit beyond the

scope of this tutorial. But it's

important because if you don't have this

then you're going to have this kind of

hanging delaying program. So I encourage

you if this is interesting to you look

into asynchronous programming. You can

actually just like plug this into chat

GBT and ask it what's going on. But

that's effectively kind of what we're

doing here. So that should allow us to

actually generate the search results.

And what we'll do is go back into main

and we'll kind of test this before we

move forward with the AI component. So

what we can do now is we can fill in

this search articles function. And to do

that, we're going to import the

newsletter service. So we're going to

say from newsletter service import. And

actually, do we want to do that? No, I

think we can just import newsletter

service like that. And actually, I just

realized when I was looking at these

that these can be class methods. Um, so

I'm just going to decorate these with

the at@ class method decorator. And I'm

going to change self to simply be cls

here. And then I'm going to do the same

thing in search web simple. So I'm going

to say at class method. And then this is

going to be cls. And I'm going to change

this to say cls where we have self

because I don't actually need access to

the instance. I just need access to the

class. And then for my generate

newsletter function, I'm just going to

make this a static method because

similarly I don't actually need access

to the instance. So I'm just going to

remove that there just to make this a

little bit cleaner code. Okay. So with

that in mind, I'm going to go back to

main. I've imported the newsletter

service. And what I'm going to do now is

from my search articles, I'm going to

return await. And then this is going to

be newsletter service dot. And this is

going to be search

web_imple.

And actually I need to change the import

again. Sorry. So I'm going to say from

newsletter service import the newsletter

service. And then this needs to be the

name of the class. So newsletter

service

dots search web simple. And then for

searching the web, all we're going to do

is pass the request topic and the

request domax articles. And I think that

should be okay. Let's see why it's

highlighting here. Okay, just cuz it's

shadowing. That's fine. And I think

that's good for this particular

function. So now that we've done that,

what we can do is we can run the code.

So I'm just going to go to my

application here. I'm going to shut it

down and I'm going to restart it. I can

just leave the ingest dev server

running. I don't need to stop this. It

can just run indefinitely. And I'm going

to go now to ingest. I'm going to

refresh. Let's just make sure that our

application is synced. It is. And what

I'm going to do is trigger a test run

again. So I'm going to go send test

event. And same thing, we're going to

have to go

newsletter

slash

generate for the data. Let's just

provide a topic of

I don't know AI agents. And let's send

the event. And now what I want to do is

look at the event and see if the search

articles worked. Uh, and it says no

search results for AI agent found. Try

modifying your search parameters. Okay,

so I'm not sure why we got that. Let me

have a look into it. Okay, so I was just

having a look at this here and the issue

that we're running into is that for the

bright data tool, it's going to look for

a zone called SER. So S E R P. So we

need to actually have the zone in bright

data called SER exactly, not anything

else. So what I did is I just created a

new zone called ZER. SER, sorry, and

deleted the other one. So what I'm going

to ask you guys to do is create this SER

zone inside of Bright Data. So again,

just do the exact same thing we did

before. Go to add SER API. Call this

SER. You shouldn't already have a name,

right? So, it's called this exactly.

Everything else will be the exact same

and you should be good to go. And then

same thing for the API key. If it's

changed, you just copy the new one and

put it in the environment variable file.

So, now that we have this, if we go back

and we try running again, it should

work. So, actually, let's go back to

ingest and let's refresh and let's go to

send test event and let's do the same

thing. So, this is going to be

newsletter

slashgenerate. And then for the data,

we're going to go topic and then AI

agent and then send the event. And let's

see if it searches this time. Okay. And

notice that this time we get a ton of

different results popping up, which

means the search was working properly,

right? We could see all of that data.

And if we go and we want to look into

the data specifically, um, we can do

that. We can just make this full screen.

And we can see in this case we have like

general search engine. As we scroll down

we have the topics. Agentic AI. We get

the title right. We get the description.

We get all the information the sources

and we're going to start using that now

in the call to our LLM. So the search is

good. Perfect. So I'm just going to

cancel that one because you know we're

getting an error on the return just cuz

we haven't finished coding this yet. And

we're going to go back to the code um

and continue. Okay. Okay, so inside of

our newsletter service, the next

function we want to write is going to be

the generate newsletter function. And to

generate the newsletter, we essentially

just need to call the AI model with a

particular prompt. So actually, let's go

to prompts. py. And what I'm going to do

is just copy in a custom prompt that I

have. And this prompt will be available

from the link in the description where I

have all of the code via GitHub. So I'm

just going to paste this. We have a

newsletter prompt. Very simple. You're

professional newsletter writer

specializing in AI and technology

content. Create a newsletter about this

topic using these articles. Provide the

articles and we provide some

instructions like write a professional

newsletter. Here's the information. Keep

it concise and engaging. Use markdown

formatting. That's it. That's our prompt

inside of prompts. py. So now we're

going to go to the newsletter service.

From the newsletter service, we're going

to import these prompts cuz we're going

to use them here. So we are going to say

from prompts

import. And then this is going to be the

newsletter system prompt as well as the

get newsletter prompt like so. Okay. So

now that we have the prompts, we can use

them to invoke the AI model. So we're

going to go into this function and we're

going to say the prompt is equal to get

newsletter prompt and we're going to

pass the topic and the search results.

Okay. Then we're going to say the

openai_appi_key

is equal to os.get get envi API key.

We're going to say the adapter is equal

to AI. OpenAI dot adapter. This is

coming from ingest. We're going to say

the OC

key is equal to the OpenAI API key. And

we're going to say the model is equal

to, and you can put whatever you want

here, but I'm going to use GPT4 Mini

because it's pretty cheap and it is

pretty fast. Okay. Then we're going to

say the response is equal to await

ctx.step.ai.infer.

Now what this is going to do is it's

going to use ingest to send the request

to open aai and it will automatically

handle anything like rate limiting or

the tokens being too large for us

automatically. So that's why we're using

this rather than using like lang chain

or the openi package. We just use ingest

and it can handle sending that call. You

also could do this with um something

like Claude for example, right? Or

another model or even O Lama, a local

model. But in our case, we're just using

OpenAI. So from here, I need to put a

name. So I'm going to say generate

newsletter content, kind of like a human

readable step name. I'm going to say the

adapter is equal to and this is going to

be the adapter that we wrote. And we're

going to say the body is equal to, and

this is going to be the information we

want to send to our model. So, we're

going to say the max tokens is equal to,

and I'm going to go with 2500. That's

the maximum output length I want. I'm

going to put the temperature,

and this is going to be equal to 0.6.

The higher temperature you go, the more

random the model will be. So, I want

this to be kind of creative and unique

because it's generating a newsletter.

I'm then going to go messages, and this

is what the model is actually going to

respond to. And for the messages, I'm

going to have a list, and I'm going to

have two messages. The first is going to

be roll system. This is a system message

that it will not directly reply to but

that it will use to know kind of how to

reply in the future. And this is going

to be the newsletter system prompt. Then

we're going to have the custom or the

user message, sorry. So we're going to

say roll user and this is the one it

will actually reply to and this is going

to be content and then the prompt that

we generated right here. Okay. So that's

actually going to generate the response

for us. And then what we're going to do

is we're going to return the response.

But to get the actual response, we need

to reference choices

zero and then messages

and then content and then dot strip.

Okay. So when we do that, it's going to

strip away any of the whites space

characters for us. Give us the actual

content of the message that it

generated. And we'll have access to that

now from this function when we call it.

Okay. So now that that's there, we

simply just need to use this from

main.py. So we're going to go to

main.py. We have our search results and

now we're going to go to the newsletter

text step and we're actually going to

call this generator. So we're going to

say await and then newsletter service.

So let's actually do this. Newsletter

service dot generate newsletter. We're

going to pass the ctx the request.topic

topic and our search results from the

previous step. Okay, so let's make this

a little bit larger so we can see it.

And you'll notice, right, that we have

newsletter text. Okay, so wait,

newsletter service.generate newsletter,

pass the context, pass the topic, and

pass the search results. Okay, so that's

all good. And then what I'm just going

to quickly do is fix something in this

function right here where we're saving

the newsletter where I just convert my

file path here to a string cuz that was

causing some problems previously and I

just want to quickly fix that up. So

effectively now we're going to search,

right? So, get the results from Bright

Data. We're then going to pass that um

to the newsletter generator. It's going

to generate that for us. And then we're

going to generate the result and save

that as a newsletter. And uh we'll have

that saved inside of the file here. In

fact, you can see we had a bunch of

newsletters saved already. However,

there wouldn't be anything there yet

because we didn't yet have the AI making

them. So, I'm just going to delete these

for now. So, let's delete. Okay. And

now, let's rerun our server and let's

test this out and see if we can get it

to work. So, I'm going to go uvun

main.py. We'll leave again the ingest

dev server running. Let's go back here.

Let's go to send test event. Let's go to

newsletter slashgenerate

for the data. We're going to pass the

topic. And let's go with cats or

something. Okay. And let's send the

event. And let's see if we look at the

runs um if this is working now. Okay.

Events runs. There we go. We can see

it's running. We can see search happened

extremely fast. In this case, 3.2 2

seconds to run. We get the results and

then it's giving me an issue here saying

AI.infer got an unexpected keyword

adapter. So I think we just spelled

adapter incorrectly. So let's just fix

the spelling. So let's go to newsletter

service. And yes,

adapter. Let's fix the spelling of that.

Let's rerun our server so that the

change takes place. Let's refresh the

ingest server. Okay. And let's go to

send test event. Same thing. Let's go

newsletter generate.

Okay. And then for the data, we're going

to go topic

cats send event. And let's see now if

this runs and if we get any problems or

if it's able to generate the newsletter

for us. Okay. So, we can see it searched

extremely fast. And now we're waiting

for this generate newsletter step to

complete. So, let's see when we get

that. This usually does take the longest

because it is relying on OpenAI to get

the response which can be slow depending

on you know how many tokens we pass it

effectively. Okay, so we're still

waiting. Hopefully this gives us a

response here in a second. Uh so we can

see what it is. Okay, so it looks like

we did get a response. If we come down

here, you can see that we get kind of

the whole newsletter generated. And then

again, we're getting this issue in

finalization where it's saying something

like messages like some key does not

exist. So let's just quickly go fix that

mistake. So we say choices zero and then

messages when this should just say

message and then content. And that

should fix that up for us. So let's

again shut this down. Restart. Go back

to ingest. Let's send another test

event. This hopefully will be the last

one. So newsletter

slashgenerate

and then data topic. Let's go with dogs

this time. Okay. send the event. Let's

wait for this to run and let's see now

um if this works. Okay, so this time of

course we got a content length exceeded

error because we passed too many tokens

to the model. So I guess we got too much

information from data there when we sent

that in. Uh so what we're going to do is

I'm just going to run this again with a

different search query and we'll test it

to see if we can fix that. So,

newsletter slashgenerate and then data

and we're going to go topic

cats. So, this is something we could

code in to make sure that we don't pass

too many characters. However, that's

going to be a little bit complex right

now. So, I'm just going to leave that as

like a potential bug that could occur.

Now, I just am doing this with cats this

time. So, let's see if when we generate

the newsletter, we don't get an error.

And this works. Okay. And for the first

time, we see completed. If I go generate

newsletter, you can see that it

generated that for me. I went to save

newsletter. Looks like that worked. And

if we go back here to the newsletters

folder, we can see that we have a cat

newsletter saved and it has emojis. It's

in markdown and everything is good.

Cool. So that means that this is

working. Now all of this is great, but I

want to show you how we can invoke this

code from code. So rather than having to

do it manually from the ingest dev

server there, I want to show you how we

could do it from a request. for example,

in Python. So you could use this from

your own custom front end or something

along those lines. So the easiest way to

do this is we're just going to go into

the run. py file and I'm just going to

copy in some content that I'm going to

leave available from the link in the

description because I don't want to

write all of it manually because a lot

of it is just these kind of random topic

ideas as you see inside of here. Okay.

So what I've done is I've import

async.io random ingest and then env. You

can see we have a bunch of random topics

that we're going to write newsletters

about. And then what I've done is

written a function called send request.

What this does is it generates a random

choice from the list of topics. And that

generates an event in ingest. Notice the

event is exactly the event type that

we've been generating ourselves. We then

try to send this request. So we say

client.end. And by saying client.end,

we're going to use the ingest client

which we've defined right here. and we

just specify what app we want to send

this request to. We then say task is

equal to and then we send this request

100 times effectively and just gather

the tasks with async.io so they run at

the same time. So a very simple script

that effectively just sends 100

different requests to ingest so that we

can test to make sure this app is

scaling and working properly. Um and

again we just simply make an event. When

that event occurs, it triggers the

function to run and then it uses bright

data, collects all the information very

quickly for us, passes it to the AI

model, and then generates the

newsletter. So, just to ensure that this

is going to work and the scalability is

there, I'm going to go UV run and then

run. py. When we do that, it's going to

send all of these requests. And then to

verify that, we can go to ingest and we

should see all of the requests popping

up in our dashboard here. We can wait a

few seconds, probably like 20, 30

seconds. So, we should start populating

all of the newsletters, and then we can

have a look at them and make sure that

they actually are valid. Okay. And you

can see that a bunch of them are popping

up as completed now as I'm kind of

sitting on this page. We can click into

them. We can see whatever has occurred,

right? And just check the status of all

of these different events. In this case,

there was an error in this one because

the content length was exceeded. So,

that could happen, right? That's fine.

That's also why we have this

observability tool. So we can check if a

mistake actually happens and we can fix

that, revert it, etc. And with ingest,

there's a lot of other features you can

do for, for example, caching responses

and um like rate limiting and retrying.

I haven't implemented that yet. That's

something I'll leave to you if you want

to look up the documentation later. So

if we go now to our code, so let's go

here and we go to the newsletters

folder, we'll see there's a bunch of

newsletters. And if I click through

them, you can see we have a bunch. So we

have, you know, climate change, crypto,

energy EV hydrogen whatever right?

And you get all of these different ones

popping up that we generated as well as

all of the different sources where it

found this information. Boom. So there

you go, guys. That is pretty much going

to wrap up this video. That was

essentially making a scalable AI agent

that is connected to the web. There was

a lot of interesting components and

features that I showed you there. And

with this knowledge, you should be able

to extrapolate this and create some

really interesting applications that

actually scale and work in a production

setting. Of course, there's a lot more

to go over to really get this pushed out

to production, like the deployment side

of things. If you want a video on that,

then definitely let me know and I'd be

happy to make that in the future.

Anyways, if you guys enjoyed this video,

make sure leave a like, subscribe to the

channel, and I will see you in the next

one.