Building Privacy-First Search With Containerized OpenSearch and Local LLMs - R. Karpe & S. Soni

Description

Building Privacy-First Search With Containerized OpenSearch and Local LLMs - Rudraksh Karpe, ZS Associates & Satyam Soni, Devtron.ai Most AI-powered search today depends on cloud services, which creates two big problems: data exposure and vendor lock-in. Sensitive information often leaves local environments, and organizations lose control over how their data is processed. This talk presents a practical alternative; combining containerized OpenSearch with local Large Language Models (LLMs) to build secure, offline-capable Retrieval-Augmented Generation (RAG) pipelines. By running models such as Qwen3 and GPT-OSS locally with tools like Ollama and LM Studio, teams can handle large-context queries (100K+ tokens) while keeping data fully under their control. Using Model Context Protocol (MCP), we’ll also show how to ground local LLMs with selective web fetches for up-to-date results without compromising privacy. Attendees of this session will gain practical insights from our real-world work with OpenSearch, helping them build their own privacy-first GenAI search systems