Update the baseline
evalgate baseline runs the eval set against the app and writes the resulting scores to a baseline file.
evalgate baselinewrote baseline.jsonIt accepts --evalset, --baseline, --app and --param, the same as run, plus --k, --tolerance, --allow-regression and --with-judge. It does not accept --report-file. See the CLI reference.
When to refresh it
Section titled “When to refresh it”Refresh in the same commit as the change that justifies it:
- You improved the app and want the new scores to be the floor.
- You changed the eval set. This one is not optional: the baseline stores the sha256 and question count of the eval set, so
evalgate runexits2until you refresh. - You changed the app on purpose in a way that trades one metric for another and a reviewer agreed.
Do not refresh it to make a red build green. The baseline is only as good as the person who last ran this command, and refreshing after a regression erases the regression.
What it keeps and what it rewrites
Section titled “What it keeps and what it rewrites”k,default_toleranceand the per-metrictolerancesof the existing file are kept. With no existing file,kis3anddefault_toleranceis0.02.--k Nand--tolerance Xchangekanddefault_tolerance. Neither touchestolerances.evalsetis rewritten with the current sha256 and question count.metricsis every metric the run produced, rounded to 6 places. Metrics with no applicable questions are left out.judge_scoreis written only with--with-judge, which needsEVALGATE_JUDGEset. Otherwise a laterrunin a CI job without the judge would fail on a missing metric.
Real check: a copy of the reference baseline edited to "k": 5 and "tolerances": {"mrr": 0.05}, then refreshed with evalgate baseline, still had "k": 5 and "tolerances": {"mrr": 0.05} afterwards. Then --k 2 --tolerance 0.05 gave "k": 2 and "default_tolerance": 0.05 with the mrr entry intact.
It refuses to lower the bar quietly
Section titled “It refuses to lower the bar quietly”If any metric in the existing baseline would be lower in the new one, or the run no longer produces it, the command writes nothing and exits 2. This real run degraded the reference app against a copy of its baseline:
evalgate baseline --baseline ref.json --param tokenizer=whitespaceevalgate: error: refusing to write a baseline that is lower than the existing baseline: hit_at_k 1.0000 -> 0.9412; mrr 0.9853 -> 0.8725; citation_correct 0.9412 -> 0.5000; citation_precision 0.9024 -> 0.5854; answer_contains_fact 0.8824 -> 0.5000. Pass --allow-regression if that is intended.To accept a lower score on purpose, say so:
evalgate baseline --allow-regressionThe flag is a speed bump, not a lock: anyone can pass it. The protection is still the diff of baseline.json in review.
Review the diff
Section titled “Review the diff”The point of committing the baseline is that the change is visible. After refreshing, read it. This is the real diff from rewriting a copy of the reference baseline with --param top_k=1 --allow-regression:
@@ -8,8 +8,8 @@ "questions": 41 }, "metrics": {- "hit_at_k": 1.0,- "mrr": 0.985294,+ "hit_at_k": 0.970588,+ "mrr": 0.970588, "citation_correct": 0.941176, "citation_precision": 0.902439, "answer_contains_fact": 0.882353,Both numbers went down. A reviewer who sees that in a pull request knows the bar was lowered and can ask why. That is the whole design: the gate cannot be weakened without a visible change to a committed file.
Improvements are never gated, so they only show up here, when someone chooses to refresh. The same goes for a change to the evalset block: a new hash or a lower question count in the diff is the signal to ask whether hard questions were removed.
A workflow that holds up
Section titled “A workflow that holds up”- Make the change to your app, or to the eval set.
evalgate run. If it fails with exit1, decide whether the drop is a bug or an accepted tradeoff. Exit2means setup, or an eval set that no longer matches the baseline.- If it is a bug, fix the app. Do not touch the baseline.
- If it is an accepted tradeoff, run
evalgate baseline --allow-regression, review the diff, and write the reason in the commit message. - If you only improved things or changed the eval set, run
evalgate baselineand review the diff, including theevalsetblock.