Skip to content

Reference demo

Everything below is the real output of the commands shown, run from the repository root against the committed baseline.json (2026-11-05, Windows, Python 3.14). The numbers are deterministic for the reference app, so you get the same ones. The tables were re-run while writing this site and match the repository’s docs/demo-regression.md, including the citation_precision row.

main stays green: the degraded settings are passed on the command line and are never committed.

Terminal window
pip install -e ".[dev]"
evalgate run # exit 0
evalgate run --param tokenizer=whitespace # exit 1
evalgate run --param top_k=1 # exit 1
evalgate run --param min_coverage=0.9 # exit 1

--param passes keyword arguments to the app factory, so the same trick works for your own app. The reference corpus is 25 passages about a made-up job-queue product, retrieved with a stdlib BM25 index and answered by picking the sentence that covers the most query terms.

Terminal window
evalgate run
MetricBaselineCurrentDeltaToleranceStatus
hit_at_k1.00001.0000+0.00000.0200PASS
mrr0.98530.9853+0.00000.0200PASS
citation_correct0.94120.9412+0.00000.0200PASS
citation_precision0.90240.9024+0.00000.0200PASS
answer_contains_fact0.88240.8824+0.00000.0200PASS
abstention_accuracy0.71430.7143+0.00000.0200PASS

No metric dropped more than its tolerance below the baseline. Exit code 0.

2. Broken tokenization (tokenizer=whitespace)

Section titled “2. Broken tokenization (tokenizer=whitespace)”
Terminal window
evalgate run --param tokenizer=whitespace

A tokenizer that splits on whitespace only, so payload? no longer matches payload.

MetricBaselineCurrentDeltaToleranceStatus
hit_at_k1.00000.9412-0.05880.0200FAIL
mrr0.98530.8725-0.11270.0200FAIL
citation_correct0.94120.5000-0.44120.0200FAIL
citation_precision0.90240.5854-0.31710.0200FAIL
answer_contains_fact0.88240.5000-0.38240.0200FAIL
abstention_accuracy0.71431.0000+0.28570.0200PASS

Regressed beyond tolerance: hit_at_k, mrr, citation_correct, citation_precision, answer_contains_fact. Exit code 1.

Note that abstention_accuracy went up. An app that finds nothing refuses everything, which looks good on that one metric. This is why the gate checks every metric in the baseline instead of one headline score.

Terminal window
evalgate run --param top_k=1
MetricBaselineCurrentDeltaToleranceStatus
hit_at_k1.00000.9706-0.02940.0200FAIL
mrr0.98530.9706-0.01470.0200PASS
citation_correct0.94120.9412+0.00000.0200PASS
citation_precision0.90240.9024+0.00000.0200PASS
answer_contains_fact0.88240.8824+0.00000.0200PASS
abstention_accuracy0.71430.7143+0.00000.0200PASS

Regressed beyond tolerance: hit_at_k. Exit code 1.

A small regression: one question out of 34 loses its gold passage, and the 0.02 tolerance still catches it. The mrr drop of 0.0147 stays inside tolerance, which is the intended behaviour of a tolerance.

Terminal window
evalgate run --param min_coverage=0.9
MetricBaselineCurrentDeltaToleranceStatus
hit_at_k1.00001.0000+0.00000.0200PASS
mrr0.98530.9853+0.00000.0200PASS
citation_correct0.94120.3529-0.58820.0200FAIL
citation_precision0.90240.4634-0.43900.0200FAIL
answer_contains_fact0.88240.3529-0.52940.0200FAIL
abstention_accuracy0.71431.0000+0.28570.0200PASS

Regressed beyond tolerance: citation_correct, citation_precision, answer_contains_fact. Exit code 1.

Retrieval is untouched, so hit_at_k and mrr pass. The answerer starts refusing questions it could answer. To learn how to read these patterns, see Read a failing report.