Getting started
You need Python 3.12 or newer. evalgate has no runtime dependencies. The only dev dependency is pytest.
-
Install from a checkout.
evalgate is not published to PyPI. Install it from the repository root:
Terminal window pip install -e ".[dev]"This puts an
evalgatecommand on your path. Check it:Terminal window evalgate --versionevalgate 0.1.0 -
Run the gate.
The repository ships a reference app, an eval set of 41 questions and a committed
baseline.json. With no flags,evalgate runuses all three:Terminal window evalgate run## evalgate: PASS| Metric | Baseline | Current | Delta | Tolerance | Status || --- | ---: | ---: | ---: | ---: | :---: || hit_at_k | 1.0000 | 1.0000 | +0.0000 | 0.0200 | PASS || mrr | 0.9853 | 0.9853 | +0.0000 | 0.0200 | PASS || citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS || citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS || answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS || abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |No metric dropped more than its tolerance below the baseline.The exit code is
0. The pathsevals/questions.jsonlandbaseline.jsonare resolved from the current directory, so run it from the repository root. -
Break the app and watch it fail.
--parampasses keyword arguments to the app factory. The reference app acceptstop_k,min_coverage,k1,bandtokenizer, so you can degrade it without editing code:Terminal window evalgate run --param top_k=1## evalgate: FAIL| Metric | Baseline | Current | Delta | Tolerance | Status || --- | ---: | ---: | ---: | ---: | :---: || hit_at_k | 1.0000 | 0.9706 | -0.0294 | 0.0200 | FAIL || mrr | 0.9853 | 0.9706 | -0.0147 | 0.0200 | PASS || citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS || citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS || answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS || abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |Regressed beyond tolerance: hit_at_k.The exit code is
1. One question out of 34 lost its gold passage, and the 0.02 tolerance caught it.
What you just used
Section titled “What you just used”| Piece | Where | What it is |
|---|---|---|
| Eval set | evals/questions.jsonl | One JSON object per line. See Eval set format. |
| Baseline | baseline.json | Scores from a known-good run, committed to git. See Baseline and tolerance. |
| App | evalgate.refapp:build | A factory returning an object with answer(question). See Plug in your own app. |
- Point the gate at your own app: Plug in your own app.
- Run it on every pull request: Run it in GitHub Actions.
- Look at four different failures side by side: Reference demo.