# SLAtech AI open answer-quality benchmark - rubric

Owner decision 2026-10-28: SLAtech only, fully open (questions, rubric, raw answers,
run script), on the live demo sites, Medical and Business first, repeated every quarter.
We do not score any other vendor's product.

## 1. What is tested

The live assistant on two public demo sites, in each of their three languages:

| Set | Site | Languages |
|---|---|---|
| med | https://med.slatech.ai/demo/clinic/{en,he,ru}/ - a clinic | en, he, ru |
| biz | https://business.slatech.ai/demo/office/{en,he,ru}/ - a business | en, he, ru |

Each language of each site is its own small site with its own facts (city, currency,
phone), so a question is checked against that language's pages only.

## 2. The questions

`questions-med.json` and `questions-biz.json`, 20 per language per site (120 in all):

- 15 **answerable** - the answer is in the visible text of one page of that site. Each
  item names the page (`source_url`) and quotes the sentence (`evidence`).
- 5 **not on the site** - nothing on that language's site answers it. The right
  behaviour is to say so and point to a person, not to invent an answer.

Questions are written the way a visitor types, in the visitor's language. They are
fixed before the run and published with the results; a question is never edited after
seeing an answer. A question found to be faulty after the run (the page does say
something, or says it differently) is marked faulty in the reviews file and left out of
the scores, with the reason - not rewritten.

## 3. How a question is asked

`tools/eval-benchmark/run.py` sends each question once to the public query endpoint,
with the demo page's own public widget key and that language's page as the page the
visitor is on - the same request the website widget makes. No conversation history, no
retries for a better answer (one retry only when the request itself fails: no HTTP 200).
The raw response is stored unchanged, one JSON line per question.

## 4. How an answer is scored

`tools/eval-benchmark/score.py`, by fixed rules, no model judging a model:

| Measure | Applies to | Passes when |
|---|---|---|
| Delivered | all | HTTP 200 and a non-empty answer |
| Correct facts | answerable | every fact group of the question appears in the answer (each group lists the accepted spellings, e.g. "120", "$120") |
| Right page cited | answerable | the page named by the question (or one listed in `also_source_urls`, which states the same fact in the same words) is among the sources returned with the answer |
| Answer in the question's language | all | letters of the question's script make up at least half of the answer's letters (85% for English); links are ignored, Latin brand and unit names inside Hebrew or Russian are allowed |
| Declined when not on the site | not on the site | the platform reports no grounding content, or the answer contains a decline phrase listed in `score.py` (DECLINE), and none of the question's `must_not_include` strings |

Median latency is reported as measured from the run machine, end to end.

## 5. Human review

Mechanical checks are strict and will sometimes be wrong both ways: a correct answer
that writes "eight in the morning" fails "8:00"; an answer that declines and then
invents a price may still contain a decline phrase. Every answer to a "not on the site"
question, and every failed answerable one, is read by a person.

A reviewer may change a verdict only in `reviews-{quarter}.json`, one entry per
question, with the new value and a one-line reason. The results table shows both
numbers - after review, and the mechanical one in brackets where they differ - so
nothing changes silently.

The DECLINE list itself changes only between runs, dated in `score.py`. On 2026-11-05,
after the 2026-Q4 run, it gained "does not provide / specify / cover" and their Hebrew
and Russian equivalents: 13 plain declines in that run carried none of the earlier
phrases. The 2026-Q4 results are scored with the extended list.

## 6. What is published

Per quarter, in `Documents/c4-benchmark/{quarter}/` and on www.slatech.ai/{lang}/eval/:
questions, raw answers, reviews, the results table, the run date, and the version of
the platform answering. Bad results are published the same way as good ones.

## 7. Repeating it

    python tools/eval-benchmark/run.py Documents/c4-benchmark/questions-med.json raw-med.jsonl
    python tools/eval-benchmark/run.py Documents/c4-benchmark/questions-biz.json raw-biz.jsonl
    python tools/eval-benchmark/score.py Documents/c4-benchmark/questions-med.json raw-med.jsonl --out med.md

Python 3.10+, no packages. The answers of a language model vary from run to run, so a
repeat will not match word for word; the scores should land close.