Evaluate AI agent response relevance using OpenAI and cosine similarity

Name: Evaluate AI agent response relevance using OpenAI and cosine similarity
Availability: InStock
Author: Jimleuk

Evaluate AI agent response relevance using OpenAI and cosine similarity preview

Open on n8n.io

$20/month : Unlimited workflows

2500 executions/month

Try free

THE #1 IN WEB SCRAPING

Scrape any website without limits

Try free

HOSTINGER

Early Deal
DISCOUNT 20%

Self-hosted n8n

Unlimited workflows - from $4.99/mo

Try free

#1 hub for scraping, AI & automation

6000+ actors - $5 credits/mo

Try free

Important notice

This workflow is provided as-is. Please review and test before using in production.

Overview

This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question.

The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here https://github.com/explodinggradients/ragas/blob/main/ragas/src/ragas/metrics/_answer_relevance.py

How it works

This evaluation works best for Q&A agents.
For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the original question using cosine similarity.
A high score indicates relevance and the agent's successful ability to answer the question whereas a low score means agent may have added too much irrelevant info, went off script or hallucinated.

Requirements

n8n version 1.94+
Check out this Google Sheet for a sample data https://docs.google.com/spreadsheets/d/1YOnu2JJjlxd787AuYcg-wKbkjyjyZFgASYVV0jsij5Y/edit?usp=sharing

Jimleuk

100 workflows

Nodes

@n8n/n8n-nodes-langchain.lmchatopenai n8n-nodes-base.evaluationtrigger n8n-nodes-base.set n8n-nodes-base.evaluation n8n-nodes-base.noop @n8n/n8n-nodes-langchain.agent @n8n/n8n-nodes-langchain.chattrigger @n8n/n8n-nodes-langchain.outputparserstructured

Complexity

advanced

Published 27 May 2025

Likes 0

View on n8n.io Download Workflow

Install path: /data/workflows/4425/4425.json

Share Your Workflow

Have a useful automation to share? Publish it and help the community.

Submit Your Template How to Submit

Related Workflows

Forecast property CAPEX and ROI weekly using Google Sheets and GPT-4o

## How It Works This workflow automates weekly capital expenditure (CAPEX) forecasting for property portfolios using a multi-agent AI architecture. It targets property managers, asset managers, and facilities finance teams who need data-driven maintenance budgeting without manual spreadsheet analysis. Three Google Sheets sources, namely: maintenance records, property data, and tenant feedback, are merged into a unified dataset. A Main Prediction Agent orchestrates three specialist sub-agents: a CAPEX Prioritizer that ranks spending needs, an ROI Simulator that models return scenarios, and a Quote Requester that fetches vendor estimates. Each agent is backed by dedicated AI models, memory, and tools including a Calculator and Financial Modeling Tool. Structured predictions are parsed, split by category, formatted, saved back to Google Sheets, and pushed to an external budgeting system via POST, delivering a fully automated, auditable CAPEX planning pipeline every week. ## Setup Steps 1. Connect Google Sheets credentials to all three read nodes and the Save Predictions node. 2. Set correct Sheet IDs for maintenance, property, and tenant feedback tabs. 3. Add Claude or OpenAI API credentials to all Chat Model nodes. 4. Configure the Financial Modeling Tool with your cost rate assumptions. 5. Replace the POST placeholder URL in Update Budgeting System with your actual endpoint. ## Prerequisites - Google Sheets account with populated maintenance, property, and tenant data - Claude or OpenAI API credentials - External budgeting system with a POST-compatible API endpoint ## Use Cases - Weekly CAPEX forecasting for multi-property real estate portfolios - Automated ROI modelling for planned renovations or equipment replacement ## Customization Add more data sources (e.g., IoT sensors, ERP exports). ## Benefits Eliminates manual CAPEX spreadsheet work with autonomous AI forecasting.

View

Turn support tickets into developer insights with OpenAI, Postgres, Slack and Jira

## Overview This workflow transforms raw support tickets into actionable developer insights using AI and data processing. It automatically detects recurring issues, identifies root causes, ranks severity, and generates a structured engineering report. By combining embeddings, clustering, and AI analysis, it helps teams prioritize bugs, understand user pain points, and take data-driven product decisions. --- ## How It Works 1. **Scheduled Trigger** - Runs automatically at a defined time (e.g., daily). 2. **Workflow Configuration** - Defines time window, similarity threshold, scoring weights, and delivery options. 3. **Fetch Feedback Data** - Retrieves recent support tickets (bugs and feature requests) from Postgres. 4. **Preprocessing** - Cleans, normalizes, and removes duplicate messages. 5. **Embedding & Clustering** - Generates embeddings using OpenAI. - Groups similar tickets using cosine similarity. 6. **Cluster Aggregation** - Combines related tickets into structured clusters. 7. **Root Cause Analysis** - AI agent analyzes clusters to identify: - Root cause - Impacted module - Severity - Debug steps - Fix direction 8. **Severity Scoring** - Calculates weighted score based on: - Frequency - Sentiment - Churn risk - Enterprise impact 9. **Report Generation** - Generates a developer-focused report including: - Executive summary - Ranked bugs - Feature requests - Risk analysis - Sprint priorities 10. **Delivery** - Sends report to Slack - Optionally creates Jira issues - Optional email delivery --- ## Setup Instructions 1. **Database Setup** - Configure Postgres credentials - Ensure `support_tickets` table exists with required fields 2. **OpenAI Configuration** - Add API key for: - Embeddings (text-embedding-3-small) - AI analysis agents 3. **Slack Integration** - Add Slack credentials - Set channel ID 4. **Email Setup (Optional)** - Configure SMTP or email service 5. **Jira Integration (Optional)** - Add Jira credentials - Set project key and issue type 6. **Customize Parameters** - Adjust: - Similarity threshold - Scoring weights - Time window 7. **Schedule Configuration** - Modify trigger timing as needed --- ## Use Cases - Product teams analyzing user feedback at scale - Engineering teams prioritizing bug fixes - SaaS companies tracking churn-related issues - Customer support insights automation - AI-driven product intelligence dashboards --- ## Requirements - OpenAI API key - Postgres database with support ticket data - Slack (optional) - Email service (optional) - Jira account (optional) - n8n instance --- ## Key Features - Automated feedback clustering using embeddings - AI-driven root cause analysis - Weighted severity scoring system - Developer-ready intelligence reports - Multi-channel delivery (Slack, Email, Jira) - Fully customizable scoring and thresholds --- ## Summary A powerful AI-driven workflow that converts raw support tickets into structured developer intelligence. It automates clustering, root cause detection, prioritization, and reporting helping teams fix the right problems faster and build better products.

View

Track LLM costs and usage across OpenAI, Anthropic, Google and more

## Installation Steps 1. Go to **Settings → n8n API** and create an API key 2. Add it as credential for the **Get Execution Data** node 3. Review model mappings in **Standardize Names** node 4. Review pricing in **Model Prices** node ## To Monitor a Workflow 1. Add **Execute Workflow** node at the end of your target workflow 2. Select this monitoring workflow 3. **Turn OFF** "Wait For Sub-Workflow Completion" 4. Pass `{ "executionId": "{{ $execution.id }}" }` as input ## Prerequisites Enable **"Return Intermediate Steps"** in your AI Agent settings for best results. ## Supported Providers **OpenAI** · **Anthropic** · **Google** · **DeepSeek** · **Meta** · **Mistral** · **xAI** · **Cohere** · **Alibaba Qwen** · **Moonshot Kimi** ### 120+ Model Variations Mapped Includes all versioned variants (e.g., gpt-4o-2024-08-06 → gpt-4o) Prices sourced from official provider pages (March 2026) ## Output Data ### Per LLM Call - Cost Breakdown (prompt, completion, total USD) - Token Metrics (prompt, completion, total) - Performance (execution time, finish reason) - Content Preview (first 100 chars I/O) - Model Parameters (temp, max tokens, timeout) - Execution Context (workflow, node, status) - Flow Tracking (previous nodes chain) ### Summary Statistics - Total executions and costs - Breakdown by model type - Breakdown by node - Average cost per call - Total execution time ## 💡 You can do anything with this data! - Store in a database for historical tracking - Send to Teams as a cost alert - Build dashboards with the summary data - Set budget thresholds and trigger warnings - Export to Google Sheets for reporting

View

Need Custom Automation?

Get help designing a custom n8n workflow that connects your stack and fits your process.

Evaluate AI agent response relevance using OpenAI and cosine similarity

Workflow preview

Important notice

Overview

This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question.

How it works

Requirements