AI-Powered Email Support Automation with RAG & Evaluation System

Description

This project is an end-to-end AI support automation workflow designed to streamline customer communication and evaluate AI response quality in real time. It combines Gmail, AI agents, vector search and automated evaluation into a single intelligent pipeline. The workflow uses two separate triggers, each with a unique purpose. The Gmail trigger handles real-time incoming emails, while the dataset trigger fetches predefined evaluation data from Google Sheets to test and measure AI performance. When an email is received through Gmail, the system extracts the subject and body and sends them to an AI support agent. The agent uses a Supabase vector database to retrieve relevant policy and FAQ information, enabling context-aware and accurate responses instead of generic replies. After retrieving the required knowledge, the AI dynamically generates a structured response including both the email subject and body. The reply is then automatically sent back to the customer through Gmail, creating a fully automated support experience. For evaluation, the dataset trigger retrieves test cases and expected answers from Google Sheets. These inputs are passed through the same AI support agent, and the generated response is compared with the expected output using another AI evaluation agent. The evaluation layer analyzes semantic similarity, factual accuracy and response quality, then assigns an accuracy score. This allows continuous monitoring and improvement of the AI system’s performance over time. By combining live email automation with automated response evaluation, this workflow reduces manual effort, improves consistency and creates a scalable AI-powered customer support system. 🔹 Tech Stack: This system integrates multiple AI and automation tools to build an intelligent support pipeline. 👽) n8n – Workflow automation and orchestration 👽) Gmail – Email receiving and response delivery 👽) OpenRouter (GPT & Gemini Models) – AI response generation and evaluation 👽) Supabase Vector Store – RAG-based knowledge retrieval 👽) OpenAI Embeddings – Semantic search and vector embeddings 👽) Google Sheets – Evaluation dataset storage and tracking 👽) JavaScript Expressions – Dynamic data handling and automation logic 🔹 Key Features: 👽) Real-time AI email support automation 👽) RAG-based contextual response generation 👽) Automated Gmail reply system 👽) Separate triggers for live support and evaluation 👽) AI-powered response quality scoring 👽) Semantic similarity and factual accuracy evaluation 👽) Google Sheets integration for test datasets 👽) Reduces manual support workload 👽) Scalable and intelligent customer support workflow