Answer quality, measured in public
On 4 November 2026 we asked two of our demo assistants 120 fixed questions in English, Hebrew and Russian and scored every answer by written rules. These are the results, the bad ones included, with every raw answer published.
Two demo sites, 60 questions each
The business demo (NorthPeak, a fictional consulting firm) and the medical demo (MedicaPlus, a fictional clinic). Each site got 20 questions per language: 15 whose answer is on the site, and 5 whose answer is not, where the right response is to say so. The questions were asked exactly as the website widget asks them.
What the assistants got right
| Measure | Business demo | Medical demo |
|---|---|---|
| Answer delivered | 60 / 60 | 60 / 60 |
| Correct facts (answer is on the site) | 29 / 45 | 35 / 45 |
| English / Hebrew / Russian | 12 / 7 / 10 of 15 | 13 / 11 / 11 of 15 |
| Right page among the sources | 22 / 45 | 30 / 45 |
| Answer in the question's language | 60 / 60 | 60 / 60 |
| Said so when the answer is not on the site | 15 / 15 | 13 / 15 |
| Median response time | 1.5 s | 1.6 s |
The weak spots, named
Contact details. Phone numbers, addresses and opening hours that sit on a contact page were often not found. The assistant then said it did not have them, rather than inventing them, but the visitor still left without an answer.
Hebrew on the business demo. 7 of 15 facts, the lowest score in the run.
Two invented answers. Asked in Russian, the medical assistant said the clinic issues medical certificates for a driving licence, and that it does not make home visits. The site says neither.
One answer from the wrong language. Asked in English for opening hours, the business assistant gave the days from the Hebrew page.
Fixed rules, and every human change shown
A script scores each answer mechanically: the expected facts must appear in it, the page holding them must be among its sources, and a question the site cannot answer must be declined. No model judges another model. A person then read every failed answer and every decline, and changed five verdicts, each with a one-line reason in the published review files.
After this run the list of phrases that count as a decline gained "does not provide", "does not specify" and their Hebrew and Russian equivalents: thirteen plain declines matched none of the old ones. The change is dated in the rubric.
We will not score a competitor's product on our own question set and publish the result as though it were neutral. Where we compare, we compare on facts you can verify against a vendor's own published terms: price, channels, languages, compliance posture.
The files
- Rubric - what is tested and how
- Questions: business, medical
- Raw answers: business, medical
- Human reviews: business, medical
- Scores per question: business, medical
- Scripts: run.py, score.py (Python 3.10+, no packages)
A language model does not answer the same way twice, so a repeat run will not match word for word; the scores should land close.
Judge it on your own content
The free tier gives you 100 answers a month against your own material. That will tell you more than any scoreboard.