Skip to content

Read a failing report

Every run prints the same Markdown report. This page walks through a real failing one.

## evalgate: FAIL
| Metric | Baseline | Current | Delta | Tolerance | Status |
| --- | ---: | ---: | ---: | ---: | :---: |
| hit_at_k | 1.0000 | 1.0000 | +0.0000 | 0.0200 | PASS |
| mrr | 0.9853 | 0.9853 | +0.0000 | 0.0200 | PASS |
| citation_correct | 0.9412 | 0.3529 | -0.5882 | 0.0200 | FAIL |
| citation_precision | 0.9024 | 0.4634 | -0.4390 | 0.0200 | FAIL |
| answer_contains_fact | 0.8824 | 0.3529 | -0.5294 | 0.0200 | FAIL |
| abstention_accuracy | 0.7143 | 1.0000 | +0.2857 | 0.0200 | PASS |
Regressed beyond tolerance: citation_correct, citation_precision, answer_contains_fact.

This came from evalgate run --param min_coverage=0.9.

ColumnMeaning
MetricA metric listed in baseline.json, in the file’s order, then any metric the run has that the baseline lacks.
BaselineThe committed score, shown to 4 decimals.
CurrentThis run’s score, or missing if the run did not produce it.
Deltacurrent - baseline, signed, rounded to 6 places then shown to 4. n/a when missing.
ToleranceThe tolerance applied to this row. All rows show the same number unless the baseline has per-metric tolerances. With --tolerance, all rows show the override.
StatusFAIL when the drop is strictly larger than the tolerance, or the metric is missing. new (not gated) for a metric the baseline does not list: it shows n/a for baseline, delta and tolerance and never changes the verdict.

The heading is PASS only if every row passes. The last line is either “No metric dropped more than its tolerance below the baseline.” or Regressed beyond tolerance: followed by the failing metric names. A reduced display precision does not change the verdict. A delta that prints as -0.0200 can pass or fail depending on the six-place value underneath.

Do not look at one number. The pattern of which metrics moved tells you which stage broke.

PatternLikely cause
hit_at_k and mrr dropRetrieval got worse: tokenizer, index, embedding model, chunking, top_k.
hit_at_k and mrr fine, citation_correct or answer_contains_fact dropRetrieval still finds the passage but the answering step lost it: prompt, model, answer selection, abstention threshold.
abstention_accuracy rises while others fallThe app is refusing more. An app that finds nothing refuses everything, which looks good on that one metric.
Only mrr drops, a littleThe right passage moved down the ranking but is still in the top k.
citation_correct and answer_contains_fact move togetherUsually the answerer abstained or picked the wrong sentence. Both need a non-empty, relevant answer.
citation_precision drops, citation_correct does notThe app cites more passages than it should, or cites something on questions it should have refused.
answer_contains_fact drops and answers got longAnswers over 600 characters never count, whatever they contain.

Applied to the report above: hit_at_k and mrr did not move, so retrieval is fine. citation_correct, citation_precision and answer_contains_fact collapsed together while abstention_accuracy went to 1.0000. The answerer is refusing questions it used to answer. In the reference app that is a coverage threshold set too high, which is exactly what min_coverage=0.9 did.

  • A small FAIL on a small set can be a single question. Check the set size. One question out of 34 is 0.0294, past the 0.02 default.
  • mrr can move within tolerance and pass while a bigger metric fails, as in the top_k=1 run.
  • If you made a deliberate tradeoff, update the baseline in the same commit and say why.

The report is an aggregate. evalgate does not print per-question results, so it will not tell you which question failed. To find out, run your app on the suspect questions directly and compare the returned retrieved_ids, cited_ids and answer to the gold fields. This is a limitation, see Limitations.