Changelog
The version in pyproject.toml is still 0.1.0. The hardening below changed behaviour, but it has no version number of its own: nothing was tagged or published after the audit, so it is listed as an entry under 0.1.0 rather than as a new release.
0.1.0, hardening after an independent audit
Section titled “0.1.0, hardening after an independent audit”Merged 2026-11-05 as pull request #7. Each finding was reproduced with a failing test before the fix, and the suite went from 87 to 146 tests.
Changed behaviour
- Exit codes.
1now means a regression and nothing else. Bad input and errors outside the verdict exit2withevalgate: error: <message>and no traceback: malformed JSONL or baseline, non-finite numbers, an eval set that changed since the baseline, a missing baseline, a bad app spec or--param, an app, factory, judge or imported module that raises, unreadable or unwritable files.EVALGATE_DEBUG=1gives the traceback. Before, an exception inside your app exited1. - Finite numbers only.
NaNand infinity are rejected in tolerances,--tolerance, baseline values,--paramvalues and metric values.--tolerance infused to make the gate accept anything. - Eval set fingerprint.
baseline.jsonrecords the sha256 of the eval set (line endings, outer spaces and blank lines normalized) and the question count.evalgate runexits2on a mismatch, and on a baseline that has no such record. evalgate baselineis safer. It keepsk,default_toleranceand per-metrictolerancesinstead of resetting them. New flags--kand--tolerancechangekand the default tolerance. It refuses, with exit2, to write a baseline that has any metric lower than the existing one or missing, unless you pass--allow-regression. It writesjudge_scoreonly with--with-judge.- Fact matching.
answer_contains_factmatches the gold fact as a whole token (7 daysno longer matches17 days,429no longer matches4290) and ignores answers longer than 600 normalized characters (MAX_ANSWER_CHARS). - Judge scores must be numbers in
[0, 1]. Anything else, includingNaNand booleans, stops the run with exit2. --paramacceptstrueandfalsein any case and rejectsnanandinf.- Extra metrics that the run produces and the baseline lacks are shown as
new (not gated)instead of being hidden. - Import path. The current directory is put on
sys.pathexplicitly, so theevalgatescript andpython -m evalgateresolve--appthe same way. The README and these docs now say that importing the module named by--apporEVALGATE_JUDGEruns its import-time code.
Added
- The
citation_precisionmetric: the share of distinct cited ids that are gold ids, 0 for an abstention on an answerable question, 1 for no citations on an unanswerable one. The committed baseline value is 0.9024. The other five baseline numbers did not move.
Not changed
- The baseline
abstention_accuracyis still 0.7143. The reference answerer makes up an answer on 2 of the 7 unanswerable questions. - An empty answer is still the only abstention.
- Determinism still holds for the reference app only. The gate can still be gamed in ways the README lists.
0.1.0, first release
Section titled “0.1.0, first release”Written 2026-11-04. It has not been published to PyPI, so “released” here means the version recorded in pyproject.toml and evalgate.__version__.
Added
evalgate run: executes an eval set against an app, compares each metric tobaseline.json, prints a Markdown report and exits0,1or2.evalgate baseline: runs the set and writes a new baseline file.- Five offline metrics:
hit_at_k,mrr,citation_correct,answer_contains_fact,abstention_accuracy. - An optional
judge_scoremetric behindEVALGATE_JUDGE, with an interface and a word-overlap stub only. - The
Appprotocol (answer(question)returninganswer,cited_ids,retrieved_ids), loaded with--app module:factoryand configured with--param. - A reference RAG app with a 25-passage corpus, a stdlib BM25 retriever and an extractive answerer, plus a 41-question eval set and a committed baseline.
- A pytest suite of 87 tests.
ci.yml, which runs the tests and the gate, andsecurity.yml.- A regression demo with real output for three degraded settings.
Known gaps in that first cut, most of them fixed by the hardening above:
evalgate baselineresetk,default_toleranceandtolerancesto their defaults.- An exception inside your app exited
1, the same as a regression. - No per-question output. This one is still true, see Limitations.