Metrics
Every metric is computed offline from what your app returned for each question. No metric calls a model unless you opt into the judge.
How scores are aggregated
Section titled “How scores are aggregated”Each question is scored individually, then each metric is the mean over the questions it applies to, rounded to 6 decimal places. Rounding means float noise never decides a gate.
| Question kind | Contributes to |
|---|---|
| Answerable | hit_at_k, mrr, citation_correct, answer_contains_fact, citation_precision |
| Unanswerable | abstention_accuracy, citation_precision |
If no question applies to a metric (for example, an eval set with no unanswerable questions), that metric has no value. It is left out of the baseline file when you write one, and it is not reported as a score. If the baseline already contains it but a later run cannot produce it, the gate fails, see Baseline and tolerance.
In the shipped set of 41 questions, 34 are answerable and 7 are unanswerable.
An app abstains when its answer, after strip(), is empty. A whitespace-only answer is an abstention. An empty answer is the only thing that counts as abstaining: a reply such as “I don’t know” is an answer.
hit_at_k
Section titled “hit_at_k”Did retrieval surface a gold passage in the top k?
For an answerable question: 1.0 if any id in the first k entries of retrieved_ids is in gold_ids, else 0.0. k is read from baseline.json (field k, default 3). It is a retrieval metric, so it does not look at the answer text. An app that retrieves the right passage and then abstains still gets a hit.
Mean reciprocal rank. How high is the first gold passage?
For an answerable question: scan the whole retrieved_ids list, not just the first k. At the first gold id found at 1-based position rank, the score is 1 / rank. If no gold id appears, the score is 0.0. A gold passage at position 1 scores 1.0, at 2 scores 0.5, at 4 scores 0.25.
citation_correct
Section titled “citation_correct”Did the app cite a gold passage, and did it actually answer?
For an answerable question: 1.0 if the app did not abstain and at least one id in cited_ids is in gold_ids; otherwise 0.0. It checks the id the app reports. It does not check whether the cited text supports the answer.
citation_precision
Section titled “citation_precision”Did the app cite only gold passages? This is what stops an app from citing every passage to collect citation_correct.
- Answerable question: the share of the distinct cited ids that are in
gold_ids. Citingp02andp07when onlyp02is gold scores0.5. An abstention, or an answer with no citations, scores0.0. - Unanswerable question:
1.0if the app cited nothing,0.0if it cited anything. Nothing is citable there.
answer_contains_fact
Section titled “answer_contains_fact”Does the answer contain the gold fact, and is the answer short enough to be an answer?
For an answerable question: 1.0 if the normalized gold_fact appears in the normalized answer as a whole token and the normalized answer is at most 600 characters (MAX_ANSWER_CHARS in metrics.py). Otherwise 0.0.
Normalization is, in order:
- Unicode NFKC normalization.
casefold().- Every run of whitespace collapsed to a single space.
- Leading and trailing whitespace stripped.
The match is bounded: the fact must not be glued to a word character before or after it, and not to a digit continued by . or ,. So 7 days does not match 17 days, 1.7 days or 7 daysX, and 429 does not match 4290. A gold fact of 256 KB matches The limit is 256 KB., the limit is 256 kb, and a sentence where the space before KB is a non-breaking space.
The length cap means that dumping a whole passage or the corpus into the answer never scores. The reference app’s longest answer is 93 characters. If your app answers in paragraphs, raise the constant in metrics.py; it is not a command-line flag.
abstention_accuracy
Section titled “abstention_accuracy”Does the app refuse when there is nothing to find?
For an unanswerable question: 1.0 if the app returned an empty answer, else 0.0. Any text, including a refusal phrase, scores 0.0.
This metric can go up when an app breaks. An app that retrieves nothing useful refuses everything, which scores 1.0 here. That is why the gate checks every metric in the baseline, not one headline number. The reference demo shows exactly this.
judge_score
Section titled “judge_score”Optional, off by default, and never used in this repository’s CI.
Set the environment variable EVALGATE_JUDGE=module:factory. The factory takes no arguments and must return an object with score(question, answer, gold_fact) -> float. For each answerable question evalgate calls it and adds a judge_score metric, the mean of the returned scores rounded to 6 places. Each score must be a real number in [0, 1]: a bool, a string, NaN, infinity or anything outside the range stops the run with exit code 2, naming the question.
evalgate ships the interface and one stub, evalgate.judge.OverlapStubJudge, which is the fraction of gold-fact words found in the answer. It is not an LLM and is not a good judge. It exists to show the shape of the interface. There is no vendor client in the package. If you wire in a real model, you own the cost, the nondeterminism and the key.
If the variable is set and the factory raises, cannot be imported, or returns something without a callable score, evalgate exits with code 2. Importing the module runs its import-time code with your permissions, so only point the variable at code you trust.
What the code does with a judge
evalgate run adds judge_score whenever EVALGATE_JUDGE is set. evalgate baseline writes it only when you also pass --with-judge, so a CI run without the variable never fails on a missing metric. If the baseline has judge_score and a later run has no judge, the row is missing and fails. If the run has a judge and the baseline does not, the row is shown as new (not gated).