Skip to content

Baseline and tolerance

baseline.json holds the scores from a run you accepted as good. You commit it. The gate compares every later run to it, so the bar only rises when someone changes the file on purpose, in a diff a reviewer can read.

This is the committed baseline of the reference app:

{
"version": 1,
"k": 3,
"default_tolerance": 0.02,
"tolerances": {},
"evalset": {
"sha256": "5245ea1577dba951dfe3be5954ad81b836ac9dac3ed2224f4980d1b63eae6f4d",
"questions": 41
},
"metrics": {
"hit_at_k": 1.0,
"mrr": 0.985294,
"citation_correct": 0.941176,
"citation_precision": 0.902439,
"answer_contains_fact": 0.882353,
"abstention_accuracy": 0.714286
}
}
FieldTypeMeaning
versionintegerFormat version. Only 1 is accepted.
kinteger, at least 1The k used for hit_at_k. Defaults to 3 if absent.
default_tolerancenumber, at least 0Tolerance for any metric not listed in tolerances. Defaults to 0.02.
tolerancesobjectPer-metric tolerances, for example {"mrr": 0.05}. Optional.
evalsetobjectsha256 of the eval set and its questions count. Written by evalgate baseline. run refuses a baseline without it.
metricsobject, non-emptyMetric name to number. Required. Every metric listed here is gated.

A baseline that is not valid JSON, has no metrics, has a non-numeric value, a negative tolerance or a bad k makes the CLI exit with code 2. So does any NaN, Infinity or -Infinity: numbers in the baseline, tolerances included, must be finite.

A baseline only means something for the eval set it was measured on. evalgate baseline stores a sha256 of the eval set file and the question count. The hash is taken after normalizing line endings, outer spaces on each line and blank lines, so those do not change it. Any change to a question, a gold field or the order of lines does.

On every evalgate run, the current eval set is hashed and compared before anything runs. If it differs, the command exits 2 and names both versions:

evalgate: error: eval set changed since the baseline was written (baseline: 6ccf635a6a7c, 4 questions; now: 1fbb5a576033, 3 questions). Review the change, then run 'evalgate baseline' and commit the result.

This is deliberate. Deleting the hard questions is the easiest way to raise a score, and now it shows up as a baseline change in review instead of passing silently. A baseline without the record is rejected the same way, with baseline does not record its eval set; re-run 'evalgate baseline' and commit it.

For each metric in the baseline:

delta = round(current - baseline, 6)
passes = round(-delta, 6) <= tolerance

A metric fails only when it drops by strictly more than its tolerance. So:

  • A drop exactly equal to the tolerance passes.
  • An improvement always passes, however large. It never fails the build.
  • A metric the baseline does not list is not gated, even if the run produces it. It still appears in the report, marked new (not gated).

Both numbers are rounded to 6 places before comparing. Without that, 0.7 - 0.6 is 0.09999999999999998 in floating point, and a tolerance of exactly 0.1 would flip on rounding noise.

The overall result is PASS only if every row passes.

In order of precedence:

  1. --tolerance X on evalgate run, which overrides every metric, including per-metric values in the file.
  2. tolerances[metric] in baseline.json.
  3. default_tolerance in baseline.json.

--tolerance must be a finite, non-negative number (--tolerance inf and nan are rejected with exit 2), and negative or non-finite values in the file are rejected too.

Terminal window
evalgate run --tolerance 0.05

If the baseline lists a metric and the current run does not produce it, that row is missing and it fails. The easiest way to pass a gate is to stop measuring what regressed, so evalgate does not allow it.

This real run used a baseline that contained judge_score while the judge environment variable was unset:

| judge_score | 0.9069 | missing | n/a | 0.0200 | FAIL |
Regressed beyond tolerance: judge_score.

The opposite case is not a failure. This is the same baseline with citation_precision removed from metrics, so the run has a metric the baseline does not:

| citation_precision | n/a | 0.9024 | n/a | n/a | new (not gated) |

A new metric never changes the verdict. It tells you there is something to add the next time you refresh the baseline.

Metrics that cannot apply to your set, such as abstention_accuracy when there are no unanswerable questions, are written to the baseline as absent. So this rule only bites when something was measured before and is not now.

The default of 0.02 is a number, not a statistical claim. evalgate uses a fixed tolerance and has no significance test.

What matters is your set size. Each question moves a metric by 1 / n. With the 34 answerable questions in the reference set, one question moves hit_at_k by about 0.029, which is more than 0.02, so one lost question fails the gate. With 200 answerable questions, one question moves it by 0.005 and a tolerance of 0.02 would let four slip through.

Decide how many questions you are willing to lose, convert that to a number and write it as the tolerance. If an LLM judge or a nondeterministic app makes scores wobble from run to run, set the tolerance above the wobble you measure, or the gate will flap and people will stop trusting it.

To set one metric more loosely, edit baseline.json by hand:

"tolerances": { "mrr": 0.05 }

evalgate baseline keeps k, default_tolerance and your per-metric tolerances. --k changes k and --tolerance changes default_tolerance; neither touches tolerances. See Update the baseline.