Evaluate RAG response accuracy with OpenAI: document groundedness metric

This n8n template demonstrates how to calculate the evaluation metric "RAG document groundedness" which in this scenario, measures the ability to provide or reference information included only in retrieved vector store documents.

The scoring approach is adapted from https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templates#pointwise_groundedness

How it works

• This evaluation works best for an agent that requires document retrieval from a vector store or similar source.
• For our scoring, we need to collect the agent's response and the documents retrieved and use an LLM to assess if the former is based off the latter.
• A key factor is to look out information in the response which is not mentioned in the documents.
• A high score indicates LLM adherence and alignment whereas a low score could signal inadequate prompt or model hallucination.

Requirements

• n8n version 1.94+
• Check out this Google Sheet for a sample data https://docs.google.com/spreadsheets/d/1YOnu2JJjlxd787AuYcg-wKbkjyjyZFgASYVV0jsij5Y/edit?usp=sharing