Limitations
evalgate is a regression gate, not an evaluation of whether your app is good. These are the places where that distinction bites.
The committed baseline is weak in one place
Section titled “The committed baseline is weak in one place”abstention_accuracyis 0.7143 in the committed baseline. The reference answerer makes up an answer on 2 of the 7 unanswerable questions. The gate only stops that number from getting worse; it does not say 0.7143 is good. Fixing the answerer would make a nicer demo but is not what the gate is for.- An empty answer is the only abstention. A real LLM app that replies “I don’t know” is scored as having answered, so it fails on unanswerable questions unless your wrapper returns an empty string in that case.
The metrics are lexical
Section titled “The metrics are lexical”- A correct answer in different words fails
answer_contains_fact. The check is a normalized whole-token match againstgold_fact. “Thirty seconds” does not match30 seconds. - A wrong answer that quotes the gold fact passes, as long as it is 600 characters or shorter. Containing the string is all it checks.
- Citation correctness checks the id, not the support.
citation_correctandcitation_precisionask whether the ids the app reports are gold ids. Neither reads the cited text to see whether it supports the answer.
The gate can still be gamed
Section titled “The gate can still be gamed”The length cap on answers, the whole-token fact match and citation_precision close the cheapest tricks, such as returning the whole corpus and citing every passage. They do not make the gate unbeatable. A determined wrapper can still optimise for lexical metrics: shape answers to contain the gold fact under 600 characters, cite exactly the right ids, or abstain with an empty string where it would be wrong. The README’s Limitations section says the same. Treat a green gate as “no regression on these checks”, not as proof of quality, and review changes to the wrapper as you would review the eval set.
The statistics are thin
Section titled “The statistics are thin”- Fixed tolerance, no significance test. The default 0.02 is just a number. With the 41-question reference set, one question moves
hit_at_kby about 0.03, which is more than the default tolerance. A tolerance only makes sense once you know your set size, see Baseline and tolerance. - Determinism holds for the reference app only. A real LLM app, or a judge backed by a model, can change between runs. If your app varies, widen the tolerance above the variation you measure or the gate will flap.
- The eval set is not checked for leakage into the app, for example questions or gold facts copied into a prompt or an index. evalgate does detect that the eval set changed since the baseline, which is a different thing.
The baseline is a human decision
Section titled “The baseline is a human decision”- It is only as good as the person who last ran
evalgate baseline. The command now keepskand tolerances, and refuses to write lower scores without--allow-regression, but anyone can pass that flag. The protection is that the change is a visible diff in a committed file, so reviewbaseline.json, including theevalsetblock. See Update the baseline.
Reporting
Section titled “Reporting”- No per-question output. The report shows aggregates. It does not tell you which question regressed, so you have to rerun your app on suspects yourself.
- The loader runs your code. Importing the module named by
--apporEVALGATE_JUDGEexecutes its import-time code with your permissions, before anything is checked. Only point them at code you trust.
- Single-turn questions only. No multi-turn conversations, no tool use, no latency or cost budgets.
- The judge is a stub.
judge_scoreis an interface and a word-overlap example, not an LLM. There is no vendor client. Judge scores must be numbers in[0, 1]. - The reference app is a toy. Its corpus is 25 short passages and its 41 questions are easy for BM25, so
hit_at_kis 1.0. The numbers show the gate works, not that this is a good RAG system.
Distribution and testing
Section titled “Distribution and testing”- Not on PyPI. Install from a checkout with
pip install -e ".[dev]". - Tested locally on Windows with Python 3.14. The CI workflow targets Python 3.12 and has run green on GitHub for
main.