Loading video player...
One of the biggest problems with most
rag systems is that we don't actually
know the sources that our AI agent is
using. Did it hallucinate? Did it pull
from irrelevant documents? And no,
citation is not enough because agents
can make that up as well. And so today,
I want to show you how to solve this
problem with a rag agent that you're
looking at right here that has human in
the loop for source validation. Users
can see the chunks that are returned
from our knowledge base, approve some
and reject others, and only then does
the agent synthesize an answer. So, we
have full control and visibility. And
boy, is this powerful. So, I built this
agent with Copilot Kit and Pideantic AI.
I'll show you what that looks like under
the hood. But first, I want to give you
a quick demo of the agent. So, the
knowledge base that this agent has
access to just contains a bunch of
information about different AI
companies. And so I'm going to restart
the conversation here and paste in a
question. I know that my face is
blocking this a little bit, but it's
just asking what is OpenAI's latest
funding. So I'll send this in. And
before we get an answer, we're going to
see the chunks appear on the lefth hand
side. And the agent recognizes like
okay, select the sources you want me to
use for the summary and then ask me to
proceed. And so I can check the box for
all the chunks that I actually want it
to include. And we can also click into
this to see exactly what we did pull
from the knowledge base. There's so much
transparency that we have here. Even
using this as an application just to
debug our rag agents and evolve our rag
agents is super powerful. So I select a
couple of chunks and I can even say what
are the sources that I have selected.
And so I'll show you that we have this
real time sync between the agent in the
backend and our front end. It knows the
three sources that we have selected. All
the other ones right now we do not have
as approved chunks. And so now I can say
yes, give me the answer. And so we'll
get the final answer, but it's based on
everything that we have curated for the
agent. This is not the kind of thing
that I've shown on my channel before.
And also, this is not the kind of thing
that is easy to engineer. Having this
real time sync with our actions in the
front end and what the agent is working
with in the back end is very powerful.
Now, as far as use cases for this,
something that I've been doing a lot now
is treating this as a rag agent harness.
And so essentially I'll take any rag
agent that I'm currently working on and
I'll plug it into this front end so I
can test its responses with different
sources selected and then also just make
sure that it's pulling the right
knowledge from the database. And you
could definitely adapt this to be more
of like an internal rag agent where
users maybe like on your team can
actually like pick and choose the
sources that the agent is searching
through. So treating the chunks return
as a recommendation that you then get to
sift through yourself if you want to
make sure that you get the most accurate
responses possible. Now for the two core
components of our tech stack that makes
this rag agent possible. We have copilot
kit for the front end and pyantic AI for
our backend agent. Copilot kit is what
enables agentic experiences as I like to
call it. So there's a video where I've
covered this on my channel already that
I'll link to right here with aentic
experiences. They also call it
generative UI. It allows our AI agent to
manipulate the state in the entire user
interface. And then also we can change
the state of the agent from the UI. So
we're just really deeply embedding the
agent in our application. And I really
do think that this is the future of SAS
and really all web development. And the
C-pilot kit team also built the AGUI
protocol. This is the secret sauce that
works under the hood to connect any
application to any AI agent. And so in
our case, we have a copilot kit app that
is connected to a paidantic AI agent.
And pretty much any agent framework you
could ever hope to use supports agui
like langraph, crewi, mra, agno. And
each one of these in their docs, they'll
tell you how to do that integration. And
so, for example, with Pantai, they have
the dedicated page here that I'll link
to in the description, of course. And it
shows you how easy it is to take any
Pantai agent and instantly make it
available as an API endpoint that is
AGUI compatible. So, you can connect it
to your C-pilot kit app. And now we get
to the fun part. How we connect this all
together because we talked about Copilot
Kit and Pideantki and Age UI. How does
this all come together? And that is with
the use agent hook which really is the
star of the show. This was released by
the co-pilot kit team just this month.
And what it does here is it gives us
this agent object. And so we have direct
access to the agent in our front end but
the agent is still running securely in a
backend API. And so now we can manage
the state for the agent. We get
real-time updates managing the
conversation history all from the front
end. And because it's using AGUI under
the hood, we can connect with any agent,
no matter how you built it or the
framework that you use. And as they put
it, we can access and interact with our
direct to LLM agents directly from React
components. And let me tell you, this is
something I always wish that I had. I
mean, yes, we can build this kind of
real-time syncing solution with our
agents manually, but it is a
fundamentally difficult problem to
engineer. And I think you'll see that as
I get into the rag agent with you. The
problem that we're solving is not easy,
but our solution is pretty simple
because we are using the use agent hook.
And so it gives us direct access to all
the information about our agent even
like if it's running or not. So we can
have real-time updates as we are waiting
for a response from the agent. We can
list out the conversation history and
manage that in our front end. Working
with the underlying state that it's
using for things like its tool calls,
even updating the state for the agent so
the next time it uses a tool call, the
state is changed. like this kind of
thing you have to do through a lot of
different API endpoints and calls if
you're not using Copilot kit and AGUI
even subscribing to agent events and so
just making your application more
ingrained with the agent and making it
feel a lot snappier because you're
constantly getting that responsiveness
as you're interacting with the agent.
All right, so now that we know all the
components in play, I want to get into
our rag agent specifically and how it
works. And of course, this is a fully
open- source GitHub repository. And so
I'll have a link to it in the
description if you want to try out this
agent for yourself. Everything is
completely free, obviously, except for
the large language model that you have
to hook into the agent in the back end.
And so, let me actually restart the
conversation here and ask a new
question. Tell me about Amazon's
investment in Enthropic. So, I want to
showcase the features again really
quickly and then get into how everything
works with the use agent hook and our
Pantic AI agent. And so first of all
what we can do here is we can select
different sources. We can deselect them
or select all. And as I'm making these
changes here it is being synced in real
time to our agent in the back end. And
so for example I'll select just one
chunk here. I'll say you know what
source do I have selected. And so it'll
say you have chunk one cloud AI wars.
There we go. All right. So it knows
right away what I'm doing in the front
end. And so now I can say okay great.
now give me a summary based on that. So
I'll send in this request and then what
we can see if I go to the backend logs
in the API we can see that for approved
chunk ids we just have a single chunk.
So it's chunk index number one for the
document titled cloud AI wars and so
right here AWS has made a significant $8
billion investment in anthropic. I wish
they would invest $8 billion in me but
yeah there we go. That is our answer
based off of just this piece of
information. And we can even click into
this to see exactly where we got that.
There we go. $8 billion investment from
Amazon Web Services in Anthropic. And if
you were to select the chunk that
doesn't have that information, it would
just straight up tell us because it is
only leveraging what we have approved
here in order to synthesize its answer.
And so think of it as like these are our
recommendations and then we get to
curate what it actually uses. And of
course, I have a nifty Excal diagram to
help you understand how this all works
at a high level. And then I'll get into
the weeds a little bit more in the code
in just a little bit. But I do want to
stay relatively high level just talking
about how everything works together. And
so first of all, we have our backend
agent. And so this is a pideantic AI
tool to search our knowledge base. If
you've done anything with Pyantic AI
before, you know how dependencies work.
These are the runtime dependencies that
we inject into our agent for all of the
tool calls and the state. This is what
we're real time syncing with our front
end using agui. This is available as
just another one of the dependencies for
our agent. And so we send in the first
query to ask the agent to search the
knowledge base and it's going to run
this tool and the tool returns a state
snapshot event. This is the event that
is streamed to the front end. So the
front end knows what chunks to display
to the user to get to that approval
stage. And so now in our front end, of
course, we have the use agent hook, the
star of the show. So this is connected
to our rag agent using copilot kit
pointing to our API endpoint for the
pyanti agent. And then whenever we call
this rag tool, we return that state
snapshot event. And so now we have the
updated state for the retrieved chunks
for us to display in a react component.
And then when the user approves the
sources in the UI, we're going to call
the set state here on the agent and
we're going to update the approved chunk
ids. And this is instantly going to be
available now in the state for the agent
in the back end. And so at this point,
the agent has just offered the sources
to approve or reject. But we are
blocking the agent from giving the final
response until the user does the
approval and then sends the next message
to the agent like we saw. And once the
user does that, then we go back to the
back end. And there's another tool that
the agent has to synthesize the
response. And so we're going to look at
the state. This is what is updated from
our front end. We're going to fetch the
approved chunk IDs. And then we're going
to generate the answer based on only the
chunks that we have approved. And so
that way we have human in the loop rag.
And that honestly just unlocks a whole
new level of reliability for our agent.
Now, I also want to spend at least a
little bit of time diving into the code
with you because I want you to
understand how easy it is to take our
use agent hook with copilot kit, our
pyantic AI agent, and connect them
together with AGUI. And of course, I
encourage you to take your AI coding
assistant and ask it questions over this
codebase if you want to understand more
how it works or especially adapt it to
your own use cases. And I do have
instructions in the readme here for how
you can get it all up and running
yourself if you want to try it out. So
the first thing that is really important
for you to understand here is how the
state works. And so these are all of the
pieces of information that we want to
sync in real time with our front end and
backend. And what you'll notice here is
that all of the types defined in our
backend. This Python script right here
is exactly the same as the types that we
have defined in the front end. And this
is important because the state is really
a contract that we have between the
front end and back end with AGUI. So the
only reason that this integration works
so seamlessly is because the front end
and the backend understand the state in
exactly the same way. Same structure,
same types, same everything. And so this
state is the dependencies for our
pyantic AI agent. And so we define our
Panti agent just like I've done a
million times on my channel already. We
get the large language model and I
support Gemini, OpenAI, Open Router,
pretty much everything here. We get our
system prompt that I have from a
separate file. And then we define our
dependencies. And this is exactly what
we have through AGUI syncing with our
front end. And then for the first tool
in our workflow, we have the one to
search the knowledge base. And so the
tools are defined very similar to what
you've done with Panti in the past if
you've worked with it before. But when
we access the state here from our
dependencies, this is what we have from
agui. This is the real time state sync
with our front end. So it instantly has
access to anything that we've been doing
in the front end where we use the use
agent hook to update the state for our
agent. And so I don't need to get into
the code for this tool too much, but we
have both hybrid and regular semantic
search. And the important thing here is
this tool is not going to give the
chunks for the agent to give its final
answer. Remember, it's just going to
list out the chunks and then send this
state snapshot event so that the front
end can take all this, display it for
the user to then approve and reject the
different chunks that we have here. And
so the system prompt for the agent also
describes this workflow as well. Like
when you first get a request from the
user, you call this tool and then you
display the chunks. You don't give the
final answer yet. We do display the
final answer for this bottom function
right here. So synthesize with sources.
Once the user has approved the sources,
then we're going to fetch that from the
state. We're going to get the list of
approved sources here. We're going to
build up this context to then return it
to the agent. So now the agent is going
to take this information to give us the
final answer. And for the icing on the
cake here, take a look at how easy it is
to take our Pantai agent and make it
AGUI compatible. We just have to call
this function and then it gives us this
app instance that we can import within
our main.py where we are setting up our
fast API endpoint. And so I just have to
mount the AGUI app and then boom, our
Panti agent is immediately available on
port 8000 thanks to AGUI and I'll show
you how we set this up the connection
here in our front end. And so first of
all in our layout we are wrapping our
entire React application with our
co-pilot kit runtime. This is what makes
it so that when we chat with our agent
in the front end, it is automatically
going to route the request to this API.
And this is where we create our copilot
kit runtime to make that connection to
localhostport 8000. That is where we are
running our paidantic AI agent. And so
now within the page.tsx, this is the
main part of our React app. There's
actually not that much code here.
Overall, think about how many API calls
you'd have to make for every single
little state sync and calls to the
agent. We don't have to do any of that
thanks to our use agent hook. Like
literally all we have to do is call use
agent. We give it the ID of the agent
that we want to connect with. And then
that gives us the instance of the agent
that we can interact with in any React
component however we want. We can read
the state and work with that any
component. We can set the state and that
is automatically reflected immediately
in our backend. Like this is our
function here to handle when the user
checks a box to approve a specific
chunk. And then we also have some
configuration which I haven't shown yet
but we can change different attributes
for how the agent performs rag under the
hood like semantic versus hybrid search.
We can reset the conversation. We can
select or deselect all sources. And so
all of these would have to be individual
API calls that we make to sync the state
or interact with the agent. But now we
don't have to do any of that. It's all
handled for us thanks to the use agent
hook. That's the power that we have
here. here. Like if I didn't have the
use agent hook, this would be hundreds,
maybe even thousands of lines of code
interacting with the different endpoints
to handle different parts of updating
the state for the agent. Like I don't
want to engineer for that. So I'm very
thankful that we have copilot kit and
agui making this possible. And also like
I mentioned earlier, copilot kit
recently had their big version 1 release
that came with the use agent hook. Now,
the use agent hook is probably the
biggest thing in this update, but there
are a lot of other really incredible
things as well. So, definitely check
this out if you're interested in
building these agentic experiences,
applications where the agent is deeply
integrated with the front end. I do this
kind of thing all the time myself. I
love Copilot Kit. I've got a direct line
of communication with the team because I
appreciate the platform so much. And of
course, I worked with them on this video
as well, getting their feedback on
things. So, thank you to them for that.
Now, the other thing I want to call out
really quickly for the 1.50 release is
their support for A2 UI. So, this is
Google's generative UI spec.
Essentially, all a generative UI spec is
is giving agents or LLMs the ability to
generate front-end components
dynamically. Hopefully, that's pretty
clear why that's such a big deal for
CopilotKit and AGUI. And so, you can
literally make it so that your agents
can generate dynamic user interfaces
based on the conversation. And that is
very exciting. And so I just want to end
by saying, let me know in the comments
what other kinds of use cases you'd want
me to build out with C-Pilot kit and
pineanti to create more agentic
experiences or maybe even building out
this rag agent more too. I really do
want to turn this into sort of like a
rag testing harness. I think that'd be
really powerful as an open- source
project. So curious what you think. So
with that, if you appreciate this video
and you're looking forward to more
things on building AI agents, I would
really appreciate a like and a subscribe
and I will see you in the next video.
One of the biggest problems with RAG systems is you never know what sources the AI Agent actually used. Did it hallucinate? Did it pull from irrelevant docs? And no, citation is not enough. In this video, I show you how to solve that by building a RAG agent with human in the loop and source validation - users see the retrieved docs/chunks, approve or reject them, and only then does the agent synthesize an answer. This is especially powerful to have as a tool in your toolbelt as you are building RAG agents and refining them! CopilotKit just dropped v1.50 with a complete AG-UI native rewrite, and it's perfect for this. I'm using their new useAgent hook for real-time state sync between a Pydantic AI backend and React frontend, plus human-in-the-loop that let users control exactly what sources get used. ~~~~~~~~~~~~~~~~~~~~~~~~~~ - If you want to build generative UI like I did for this RAG agent, check out CopilotKit: https://copilotkit.ai ~~~~~~~~~~~~~~~~~~~~~~~~~~ - In partnership with AWS and Kiro, I'm hosting an AI Coding Hackathon in January with a $17,000 prize pool! See details and register for free here: https://dynamous.ai/kiro-hackathon - The Dynamous Agentic Coding Course is now FULLY released - learn how to build reliable and repeatable systems for AI coding: https://dynamous.ai/agentic-coding-course - RAG Agent with CopilotKit + AG-UI + Pydantic AI: https://github.com/coleam00/human-in-the-loop-rag-agent - Pydantic AI AG-UI Docs: https://ai.pydantic.dev/ui/ag-ui/ - CopilotKit useAgent Hook Docs: https://docs.copilotkit.ai/direct-to-llm/guides/use-agent-hook - CopilotKit A2UI Launch: https://docs.copilotkit.ai/whats-new/a2ui-launch ~~~~~~~~~~~~~~~~~~~~~~~~~~ 00:00 - Introducing the Human in the Loop RAG AI Agent 02:48 - Our Tech Stack - CopilotKit, Pydantic AI, and AG-UI 04:13 - The New useAgent Hook (Star of the Show) 06:02 - RAG Agent with Human Source Validation in Action 07:58 - Excalidraw Diagram for the Agent Architecture 10:11 - Quick Code Dive to Understand the RAG Flow 15:24 - New CopilotKit Release + Google A2UI Integration 16:30 - Final Thoughts ~~~~~~~~~~~~~~~~~~~~~~~~~~ Join me as I push the limits of what is possible with AI. I'll be uploading videos weekly - at least every Wednesday at 7:00 PM CDT!