evalgate
Prompt and retriever changes tend to merge with “looks fine on the three examples I tried”. evalgate gives answer quality the same check a test suite gives code. This is the real output of one degraded run of the bundled reference app, evalgate run --param tokenizer=whitespace, exit code 1:
## evalgate: FAIL
| Metric | Baseline | Current | Delta | Tolerance | Status || --- | ---: | ---: | ---: | ---: | :---: || hit_at_k | 1.0000 | 0.9412 | -0.0588 | 0.0200 | FAIL || mrr | 0.9853 | 0.8725 | -0.1127 | 0.0200 | FAIL || citation_correct | 0.9412 | 0.5000 | -0.4412 | 0.0200 | FAIL || citation_precision | 0.9024 | 0.5854 | -0.3171 | 0.0200 | FAIL || answer_contains_fact | 0.8824 | 0.5000 | -0.3824 | 0.0200 | FAIL || abstention_accuracy | 0.7143 | 1.0000 | +0.2857 | 0.0200 | PASS |
Regressed beyond tolerance: hit_at_k, mrr, citation_correct, citation_precision, answer_contains_fact.You keep two things in your repo: an eval set in JSONL (questions plus the passage ids and fact a correct answer must contain) and a baseline.json of scores. evalgate run executes the set against your app, compares each metric with the baseline, prints a Markdown table, and exits 1 if any metric dropped by more than its tolerance. Exit 2 is reserved for bad input or an error in your app, so a crash never looks like a regression.
It has no runtime dependencies, makes no network calls in its default path, and gives the same result every time. That is deliberate: a gate that needs an API key or varies between runs gets ignored.
Where to go next
Section titled “Where to go next”- New here? Getting started installs it and runs the gate against the reference app in a few minutes.
- Have an app? Plug in your own app shows the three-key contract and a runnable example.
- Want the rules? Metrics says exactly how each score is computed, and Baseline and tolerance says exactly when a metric fails.
- Deciding whether to trust it? Read the limitations first. The metrics are lexical and the bundled app is a toy.