Decisions
This page mirrors docs/decisions.md in the repository, which is the source of truth. Each entry is a decision, the alternatives considered, and the reason. Entries are dated 2026-11-04, except the last one and an amendment, which are 2026-11-05.
Stdlib only, no vector store, no LLM in the default path
Section titled “Stdlib only, no vector store, no LLM in the default path”Alternatives: sentence-transformers or an embeddings API for retrieval, an LLM for answering and judging.
Why: the point of the project is the gate, not the RAG app. Anything that needs a key or downloads a model makes CI flaky, slow or paid, and a flaky gate gets ignored. BM25 and an extractive answerer are enough to produce metrics that can actually move when something breaks. The cost is that the reference app is a toy.
The gate fails only on drops larger than the tolerance, strictly
Section titled “The gate fails only on drops larger than the tolerance, strictly”Alternatives: fail on any change, fail on any drop, compare against a fixed absolute threshold instead of a baseline.
Why: a committed baseline means the bar rises only when someone updates it on purpose, in a reviewable diff. Strictly greater means a drop exactly equal to the tolerance passes, which is the one rule that is easy to state and test. Improvements never fail, they just show up in the diff of baseline.json when someone refreshes it.
Round to 6 places before comparing
Section titled “Round to 6 places before comparing”Alternatives: compare raw floats, use fractions.Fraction, add an epsilon.
Why: 0.7 - 0.6 is 0.09999999999999998 in floats, so a tolerance of exactly 0.1 would flip on rounding noise. Rounding both metrics and deltas to 6 places is two lines, and the baseline file stores the same rounded values.
A baseline metric missing from the current run fails
Section titled “A baseline metric missing from the current run fails”Alternatives: skip it with a warning.
Why: the easiest way to pass a gate is to stop measuring the thing that regressed. Metrics that are not applicable (no unanswerable questions in the set) are stored as absent in the baseline, so this only bites when something was measured before and is not now.
Abstention means an empty answer string
Section titled “Abstention means an empty answer string”Alternatives: an explicit abstained flag in the app result, a magic refusal phrase.
Why: it keeps the App protocol at three keys and a user’s app can adopt it without learning a flag. A magic phrase would have to be matched against free text, which breaks the first time someone rewords it.
Exit codes 0, 1, 2
Section titled “Exit codes 0, 1, 2”Alternatives: one non-zero code for everything.
Why: a bad JSONL line or a missing baseline is not a quality regression, and a CI log should say which one happened. 1 is a regression, 2 is bad input.
Amended 2026-11-05: 2 also covers app, factory, judge and import errors and unreadable or unwritable files. Before that an app exception gave a traceback and exit 1, which a CI job reads as “quality dropped”. The traceback is behind EVALGATE_DEBUG=1. See Exit codes.
Apps are plugged in as module:factory plus --param
Section titled “Apps are plugged in as module:factory plus --param”Alternatives: a config file, subclassing a base class, a plugin entry point.
Why: it needs no registry and no config format, and --param top_k=1 is what makes the regression demo reproducible without editing code.
The LLM judge is an interface and a stub, nothing more
Section titled “The LLM judge is an interface and a stub, nothing more”Alternatives: ship an OpenAI or Anthropic client.
Why: a vendor call cannot be tested in CI without a key, and shipping untested network code in a tool whose job is trust would undercut it. The stub is a word overlap score and says so. It is off unless EVALGATE_JUDGE is set.
setuptools and a plain console script instead of a uv project
Section titled “setuptools and a plain console script instead of a uv project”Alternatives: uv with a lockfile, hatchling.
Why: there are no runtime dependencies, so a lockfile would only pin pytest. pip install -e ".[dev]" works everywhere the CI runner and a reviewer already have Python.
An independent audit found the gate was easy to game, so I tightened it
Section titled “An independent audit found the gate was easy to game, so I tightened it”2026-11-05. Findings, each reproduced with a failing test before the fix:
- Non-finite numbers.
NaNand infinity passed through tolerances, and--tolerance infmade the gate accept anything. Tolerances, baseline numbers,--paramvalues and metric values must now be finite, andbaseline.jsonis parsed so thatNaNandInfinitytokens are rejected. - Citing everything. An app that cited every passage and returned the whole corpus passed, and even raised
answer_contains_fact. I addedcitation_precision(distinct cited ids that are gold, 0 for an abstention on an answerable question, 1 for no citations on an unanswerable one) and a cap of 600 normalized characters on answers that count foranswer_contains_fact. Alternative: a cap relative to the gold fact length. A plain constant is easier to explain and to override. The baseline was regenerated:citation_precisionis 0.9024, the other numbers did not move. - Deleting hard questions. The baseline now stores a sha256 of the eval set and the question count. A changed eval set exits 2 and needs
evalgate baseline. Alternative: warn and continue. I chose exit 2 because deleting hard questions is the easiest way to raise a score. A baseline without the record is rejected the same way. - Quiet baseline rewrites.
evalgate baselinekeepskand tolerances, refuses lower scores without--allow-regression, and writesjudge_scoreonly with--with-judge, so a CI run without the judge env var does not fail on a missing metric. - Substring fact match. The gold fact match is token-bounded (
7 daysno longer matches17 days). - Silent extra metrics. A metric the run has and the baseline lacks is printed as
new (not gated). - Import path. The cwd is put on
sys.pathexplicitly soevalgateandpython -m evalgateresolve--appthe same way. Importing the--appandEVALGATE_JUDGEmodules runs their import-time code, which the README states.
What I did not change: the abstention baseline is still 0.7143 (the reference answerer fabricates an answer on 2 of 7 unanswerable questions). Fixing the answerer would make a better demo but is not the point of the gate. An empty answer is still the only abstention, so a real LLM app that says “I don’t know” needs a wrapper. Determinism is a property of the reference app, not of the tool.