AI Quality & Evaluation
Chatbot Compare
Side-by-side evaluation of customer-support chatbots
Overview
Chatbot Compare runs 2 to 4 chatbots through the same test suite and shows which one is best, and why. Any chatbot with an HTTP API can be plugged in, and every answer is checked for accuracy, safety, hallucinations, bias and speed.
The challenge
Choosing between chatbots, or between models for one chatbot, usually comes down to a few hand-picked conversations. That hides hallucinations, unsafe answers and unfair treatment of different customers.
Key features
117 shared test templates
Price lookups, order tracking, follow-ups, Hindi, prompt injection, guardrails, out-of-scope, hallucination and bias, filled in with each bot's own store data.
Three checks per answer
Exact fact checks, an AI judge grading correctness, helpfulness and tone, and response-time measurement.
Hallucination and bias detection
Flags any invented price, spec or policy, and compares answers given to customers of different gender, age, region, religion and more.
Fair scoring
Bots are ranked only on cases all of them were scored on, with bootstrap confidence ranges and a second judge from another model family.
Gray-box and white-box tests
Checks which agent or tool handled a request and what data was retrieved, separating retrieval failures from generation failures.
Blind arena
People chat with two anonymous bots side by side and vote; ratings use the Bradley-Terry model on the Elo scale.
How it works
- Register the bots to compare by URL; the tool works out how to talk to each one.
- Pick a test suite: store-specific cases with each bot's facts, or 42 generic cases that work on any bot.
- Run the suite; every reply is fact-checked, graded by an AI judge and timed.
- Read the scorecard: headline scores, per-skill charts, head-to-head results and the failed cases with evidence.
Technology
- Python
- FastAPI
- LLM-as-judge
- NVIDIA NIM
- Gemini
- Amazon Bedrock
- Ollama
Have a similar challenge?
Talk to our team about building AI, automation or Odoo solutions for your business.