Reference demo
Everything below is the real output of the commands shown, run from the repository root against the committed baseline.json (2026-11-05, Windows, Python 3.14). The numbers are deterministic for the reference app, so you get the same ones. The tables were re-run while writing this site and match the repository’s docs/demo-regression.md, including the citation_precision row.
main stays green: the degraded settings are passed on the command line and are never committed.
pip install -e ".[dev]"evalgate run # exit 0evalgate run --param tokenizer=whitespace # exit 1evalgate run --param top_k=1 # exit 1evalgate run --param min_coverage=0.9 # exit 1--param passes keyword arguments to the app factory, so the same trick works for your own app. The reference corpus is 25 passages about a made-up job-queue product, retrieved with a stdlib BM25 index and answered by picking the sentence that covers the most query terms.
1. Unchanged app
Section titled “1. Unchanged app”evalgate run| Metric | Baseline | Current | Delta | Tolerance | Status |
|---|---|---|---|---|---|
| hit_at_k | 1.0000 | 1.0000 | +0.0000 | 0.0200 | PASS |
| mrr | 0.9853 | 0.9853 | +0.0000 | 0.0200 | PASS |
| citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS |
| citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS |
| answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS |
| abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |
No metric dropped more than its tolerance below the baseline. Exit code 0.
2. Broken tokenization (tokenizer=whitespace)
Section titled “2. Broken tokenization (tokenizer=whitespace)”evalgate run --param tokenizer=whitespaceA tokenizer that splits on whitespace only, so payload? no longer matches payload.
| Metric | Baseline | Current | Delta | Tolerance | Status |
|---|---|---|---|---|---|
| hit_at_k | 1.0000 | 0.9412 | -0.0588 | 0.0200 | FAIL |
| mrr | 0.9853 | 0.8725 | -0.1127 | 0.0200 | FAIL |
| citation_correct | 0.9412 | 0.5000 | -0.4412 | 0.0200 | FAIL |
| citation_precision | 0.9024 | 0.5854 | -0.3171 | 0.0200 | FAIL |
| answer_contains_fact | 0.8824 | 0.5000 | -0.3824 | 0.0200 | FAIL |
| abstention_accuracy | 0.7143 | 1.0000 | +0.2857 | 0.0200 | PASS |
Regressed beyond tolerance: hit_at_k, mrr, citation_correct, citation_precision, answer_contains_fact. Exit code 1.
Note that abstention_accuracy went up. An app that finds nothing refuses everything, which looks good on that one metric. This is why the gate checks every metric in the baseline instead of one headline score.
3. Smaller top_k (top_k=1)
Section titled “3. Smaller top_k (top_k=1)”evalgate run --param top_k=1| Metric | Baseline | Current | Delta | Tolerance | Status |
|---|---|---|---|---|---|
| hit_at_k | 1.0000 | 0.9706 | -0.0294 | 0.0200 | FAIL |
| mrr | 0.9853 | 0.9706 | -0.0147 | 0.0200 | PASS |
| citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS |
| citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS |
| answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS |
| abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |
Regressed beyond tolerance: hit_at_k. Exit code 1.
A small regression: one question out of 34 loses its gold passage, and the 0.02 tolerance still catches it. The mrr drop of 0.0147 stays inside tolerance, which is the intended behaviour of a tolerance.
4. Answerer too picky (min_coverage=0.9)
Section titled “4. Answerer too picky (min_coverage=0.9)”evalgate run --param min_coverage=0.9| Metric | Baseline | Current | Delta | Tolerance | Status |
|---|---|---|---|---|---|
| hit_at_k | 1.0000 | 1.0000 | +0.0000 | 0.0200 | PASS |
| mrr | 0.9853 | 0.9853 | +0.0000 | 0.0200 | PASS |
| citation_correct | 0.9412 | 0.3529 | -0.5882 | 0.0200 | FAIL |
| citation_precision | 0.9024 | 0.4634 | -0.4390 | 0.0200 | FAIL |
| answer_contains_fact | 0.8824 | 0.3529 | -0.5294 | 0.0200 | FAIL |
| abstention_accuracy | 0.7143 | 1.0000 | +0.2857 | 0.0200 | PASS |
Regressed beyond tolerance: citation_correct, citation_precision, answer_contains_fact. Exit code 1.
Retrieval is untouched, so hit_at_k and mrr pass. The answerer starts refusing questions it could answer. To learn how to read these patterns, see Read a failing report.