🧠 Auto-Evaluating Chatbots with GenAI: The Pipeline, The Prompts, and The Proof

Chatbots are getting scary good — but evaluating them? That’s still a pain. BLEU and ROUGE scores feel like trying to judge a movie by its subtitles. Human evaluation is time-consuming, inconsistent, and honestly… nobody has time for that.

So here’s the question I tackled in this project:
Can we let an LLM evaluate other LLMs?

Spoiler: Yep. And it’s shockingly effective.

The Big Idea: LLM Rating LLM 💡

We built an Auto-Eval system using Google’s Gemini 2.0 Flash model to rate chatbot responses on:

✅ Relevance – Does it actually answer the question?

✅ Helpfulness – Is the answer useful or just fluff?

✅ Clarity – Can a human actually understand it?

✅ Factual Accuracy – Is it hallucinating or nah?

And we didn’t just invent our own data — we pulled real conversations from the OpenAssistant dataset (OASST1). These are crowdsourced human-assistant chats, so it’s the real deal.

Setup: Let’s Get Nerdy ⚙️

Step 1: Load the Dataset

We used Hugging Face’s datasets library to load OpenAssistant’s training data and converted it to a Pandas DataFrame.

from datasets import load_dataset

oasst = load_dataset("OpenAssistant/oasst1")
df = oasst["train"].to_pandas()

Step 2: Extract Prompt-Response Pairs

We filtered English conversations and merged assistant replies with the prompts that triggered them.

df = df[df['role'].isin(['prompter', 'assistant'])][['message_id', 'parent_id', 'text', 'role', 'lang']]
df = df[df['lang'] == 'en']

merged = df.merge(df, left_on="parent_id", right_on="message_id", suffixes=("_reply", "_prompt"))
merged = merged[['text_prompt', 'text_reply']].rename(columns={'text_prompt': 'prompt', 'text_reply': 'response'})

Prompt Engineering + Gemini Setup 🤖

We used few-shot prompting to make Gemini behave like an evaluator and return structured scores (in JSON format).

Here’s the eval prompt we send:

def build_eval_prompt(prompt, response):
    return f"""
You are an evaluator. Rate this response to a user prompt.

Rate from 1 to 5 on:
- Relevance
- Helpfulness
- Clarity
- Factual accuracy

Return ONLY valid JSON:
{{
  "relevance": X,
  "helpfulness": X,
  "clarity": X,
  "factuality": X
}}

Prompt: {prompt}
Response: {response}
"""

And then we just hit Gemini with that:

import google.generativeai as genai

genai.configure(api_key="YOUR_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")

response = model.generate_content(build_eval_prompt(prompt, response))

Running the Evaluation Loop 🧪

We ran the model on a sample of 15 prompt-response pairs and parsed the scores:

ratings = []

for _, row in sampled.iterrows():
    try:
        res = model.generate_content(build_eval_prompt(row['prompt'], row['response']))
        json_block = res.text[res.text.find('{'):res.text.rfind('}')+1]
        score = json.loads(json_block)
        score.update(row)
        ratings.append(score)
    except Exception as e:
        print("Error:", e)

Boom. Now we have an LLM scoring LLMs. Matrix-style.

Visualizing the Scores 📊

We saved everything to a CSV and used a Seaborn boxplot to get the vibes:

import seaborn as sns
import matplotlib.pyplot as plt

ratings_df = pd.DataFrame(ratings)
sns.boxplot(data=ratings_df[['relevance', 'helpfulness', 'clarity', 'factuality']])
plt.title("LLM Auto-Eval Score Distribution")
plt.show()

And the results? Pretty solid. Some outliers, but Gemini gave reasonable scores across all four dimensions.

Takeaways 🔍

✅ This works. Gemini can evaluate chatbot responses consistently.

🎯 It scales. No need to bug your friends to rate 200 replies.

🤖 Model comparisons just got easier. Want to compare GPT vs Claude vs Mistral? Auto-eval it.

What’s Next? 🛣️

📈 Add more examples and multiple models for A/B testing.

🤯 Detect hallucinations automatically.

🧑‍⚖️ Compare LLM vs human evaluations — who rates better?

🧪 Try It Yourself

Want to peek under the hood or run it with your own data?

👉 Check out the full notebook on Kaggle

Clone it, tweak it, break it (just don’t blame me 😅).

P.S : This post was rated 5/5 on clarity by my cat. And 2/5 on factuality by my anxiety.