Loading video player...
Hey folks, it's Sydney from the Python
open source team at Langchain, and I'm
super excited to bring you our next
edition of our middleware series. So
model outages can be incredibly
unpredictable, but luckily with
Langchain's new model fallback
middleware, the reliability of your
application doesn't have to be. Whatever
the reason may be that your model calls
are no longer working, that could be a
provider outage or an API quota
exhausted, lingchain's model fallback
middleware can help you fall back to a
safe and functional model to keep your
application running. Let's take a look
at our new docs to check this out.
All right, so we're looking at our new
middleware docs. We have an overview
page, a list of all of our built-in
middlewares, both provider agnostic and
provider specific middlewares, and then
also a whole page on how you can create
your own custom middlewares. So, we're
looking at the model fallback
middleware, which as mentioned can help
you build more resilient agents that can
handle model outages, falling back when
you run out of API credits for a given
model. And then specifically one benefit
of using lingchain here is that you can
fall back to models that have different
providers. So you can have a primary
open AI model and then fall back to
anthropic model etc. So specifying a
fallback model is as simple as providing
your primary model to create agent and
then adding this model fallback
middleware with any [snorts] number of
fallbacks that you'd like. Let's take a
look at this in action. So here I have a
super simple script. I'm using lang
chain's create agent primitive to
simulate model failure. I'm just using
anthropic model that doesn't exist and
then I also have a fallback model that
doesn't exist and then our final model
that we expect to work which is GPT4
mini from openAI. So let's demo this in
studio. So we have a super simple agent
graph and I'm just going to ask the
model what's 15 * 23.
We can look at our linksmith trace view
here to get a good idea of what's
happening. So we see the model is called
with our human message input. What's 15*
23? We see that initial call to our
chatanthropic model fails. It's because
we have that typo. But this could be in
real life like an outage or an API
credit limit. Then we see our next
fallback model also simulated with a
typo here fails. And then finally, we
fall back to our last GPT 40 mini model
and get the expected result. Great. So
this has been a quick demo of
Lingchain's model fallback middleware.
We're super excited to see the
resilience that you build into your
applications with this.
Learn about how to use LangChain's new model fallback middleware to add resilience to your applications. Automatically switch between models and even providers when an API call fails with just a few lines of code. Our example simulates model failures to demonstrate how the fallback middleware can be used to ensure an agent runs regardless of provider outages or exceeded API quotas. Middleware docs: https://docs.langchain.com/oss/python/langchain/middleware/built-in#model-fallback Code: https://gist.github.com/sydney-runkle/3aee40d00a7b38c6c197bff9b2eed1a6