Skip to content

Exit codes

A CI log should say which kind of failure happened, so evalgate uses three codes.

CodeMeaningOutput
0Every baseline metric is within tolerance, or evalgate baseline wrote the file.Report on stdout (run) or wrote <path> (baseline).
1Regression, and nothing else: at least one metric dropped by more than its tolerance or is missing.Report on stdout.
2Bad input, or an error outside the gate’s verdict. No verdict was reached.evalgate: error: <message> on stderr.

Exit 1 only ever means a regression. Anything that goes wrong outside the comparison, including an exception from your own app, exits 2 with a one-line message and no traceback.

These messages are real output.

ProblemMessage
Malformed JSONL lineevalgate: error: bad.jsonl:1: invalid JSON (Expecting ',' delimiter)
Duplicate idevalgate: error: dup.jsonl:2: duplicate question id 'q1' (first seen on line 1)
Eval set unreadableevalgate: error: cannot read eval set missing.jsonl: No such file or directory
Eval set not UTF-8evalgate: error: cannot read eval set bin.jsonl: not valid UTF-8 (invalid start byte at byte 0)
Missing baselineevalgate: error: baseline not found: nope.json. Create one with 'evalgate baseline' and commit it.
NaN in the baselineevalgate: error: baseline: NaN is not allowed, numbers must be finite
Baseline has no eval set recordevalgate: error: baseline does not record its eval set; re-run 'evalgate baseline' and commit it
Eval set changedevalgate: error: eval set changed since the baseline was written (baseline: 6ccf635a6a7c, 4 questions; now: 1fbb5a576033, 3 questions). Review the change, then run 'evalgate baseline' and commit the result.
App spec without a colonevalgate: error: bad app spec 'shop', expected module:factory
Module cannot be importedevalgate: error: cannot import 'nosuch': ModuleNotFoundError: No module named 'nosuch'
Attribute missing from moduleevalgate: error: module 'shop' has no attribute 'nosuch'
Malformed --paramevalgate: error: bad --param 'top_k', expected key=value
Non-finite --paramevalgate: error: bad --param value 'nan': numbers must be finite
Factory rejects a parameterevalgate: error: factory 'shop:build' rejected params {'nope': 1}: got an unexpected keyword argument 'nope'
Your app raisesevalgate: error: question q1: app raised RuntimeError: model endpoint timed out
App returned the wrong shapeevalgate: error: question q1: app result: 'cited_ids' must be a list of strings
Judge returns a bad scoreevalgate: error: question q1: judge returned 1.5, expected a number in [0, 1]
Bad --toleranceevalgate: error: --tolerance must be a finite number, not negative
baseline would lower a metricevalgate: error: refusing to write a baseline that is lower than the existing baseline: hit_at_k 1.0000 -> 0.9412; ... Pass --allow-regression if that is intended.
--with-judge without a judgeevalgate: error: --with-judge needs EVALGATE_JUDGE=module:factory
Report file not writableevalgate: error: cannot write report nodir/r.md: No such file or directory

A flag that argparse itself rejects, such as --tolerance abc or a missing command, prints a usage message and also exits 2.

The report file error is raised after the report was printed to stdout, so the table is still in the log.

Before the hardening release, an exception inside answer() produced a traceback and exit 1, which a CI job reads as “quality dropped”. Now evalgate catches it, names the question and exits 2:

evalgate: error: question q1: app raised RuntimeError: model endpoint timed out

The same goes for a factory that raises, a judge that raises, and an import-time error in the module you point --app or EVALGATE_JUDGE at.

Set EVALGATE_DEBUG=1 and the exception propagates instead of being summarised, so you see the full traceback. That is a debugging switch, not a CI setting: with it on, an escaped exception makes Python itself exit 1, so the code no longer tells you “regression” from “crash”.

Any CI system treats non-zero as failure, so you rarely need to distinguish 1 from 2 in logic. The distinction is for people reading the log: 2 means fix your setup or your app, 1 means your change made answers worse. See Run it in GitHub Actions.