Skip to content

CLI reference

evalgate has two commands, run and baseline. You can invoke it as evalgate (the console script installed by pip) or python -m evalgate. Both put the current directory on sys.path explicitly, so --app mymodule:build resolves the same way either way. See Plug in your own app.

evalgate --help
usage: evalgate [-h] [--version] {run,baseline} ...
CI ship gate for RAG and LLM apps.
positional arguments:
{run,baseline}
run run the eval set and gate against the baseline
baseline run the eval set and write a new baseline
options:
-h, --help show this help message and exit
--version show program's version number and exit

evalgate --version prints evalgate 0.1.0. A command is required. Running evalgate alone prints a usage error and exits 2.

The --evalset, --baseline, --app and --param flags exist on both commands. --tolerance exists on both too, but means something different: on run it overrides every tolerance, on baseline it sets the default tolerance that is written.

Run the eval set against the app, compare to the baseline, print the report.

evalgate run --help
usage: evalgate run [-h] [--evalset EVALSET] [--baseline BASELINE] [--app APP]
[--param KEY=VALUE] [--tolerance TOLERANCE]
[--report-file REPORT_FILE]
options:
-h, --help show this help message and exit
--evalset EVALSET
--baseline BASELINE
--app APP module:factory (default: reference app)
--param KEY=VALUE keyword argument for the app factory, repeatable
--tolerance TOLERANCE
override every tolerance in the baseline
--report-file REPORT_FILE
also write the Markdown report here
FlagDefaultBehaviour
--evalset PATHevals/questions.jsonlThe eval set. Relative to the current directory.
--baseline PATHbaseline.jsonThe baseline to compare against. Missing file exits 2.
--app MODULE:FACTORYevalgate.refapp:buildThe app factory, see Plug in your own app.
--param KEY=VALUEnoneKeyword argument passed to the factory. Repeatable. Values become int, then float, then a boolean (true or false in any case), else stay strings. nan and inf are rejected. A pair without = or with an empty key exits 2.
--tolerance Xfrom the baselineA finite float that replaces every tolerance, including per-metric values. Negative, nan or inf exits 2. A non-number is rejected by argument parsing, also exit 2.
--report-file PATHnoneAlso write the Markdown report to this path, as UTF-8. The report still goes to stdout.

run reads k from the baseline (default 3), so hit_at_k is computed with the same k the baseline used. Before running the app it checks that the eval set matches the sha256 and question count stored in the baseline, and exits 2 if it does not.

Exit codes: 0 pass, 1 regression only, 2 bad input or an error in the app, factory, judge or files. See Exit codes.

Run the eval set and write a new baseline.

evalgate baseline --help
usage: evalgate baseline [-h] [--evalset EVALSET] [--baseline BASELINE]
[--app APP] [--param KEY=VALUE] [--k K]
[--tolerance TOLERANCE] [--allow-regression]
[--with-judge]
options:
-h, --help show this help message and exit
--evalset EVALSET
--baseline BASELINE
--app APP module:factory (default: reference app)
--param KEY=VALUE keyword argument for the app factory, repeatable
--k K k for hit_at_k (default: keep the existing baseline's
k, else 3)
--tolerance TOLERANCE
default tolerance (default: keep the existing one,
else 0.02)
--allow-regression write the baseline even if a metric is lower than the
existing one
--with-judge run the EVALGATE_JUDGE judge and store judge_score

--evalset, --baseline, --app and --param have the same meaning and defaults as in run. Here --baseline is the file that gets written, and the one it reads to keep the existing settings. On success it prints wrote <path> and exits 0.

FlagDefaultBehaviour
--k Nexisting baseline’s k, else 3The k for hit_at_k. Must be at least 1.
--tolerance Xexisting default_tolerance, else 0.02The default tolerance to write. Must be finite and not negative. Per-metric tolerances in the existing file are kept as they are.
--allow-regressionoffWithout it, the command refuses (exit 2) to write a baseline in which any metric is lower than in the existing file, or is missing from this run. With it, the lower numbers are written.
--with-judgeoffRuns the judge from EVALGATE_JUDGE and writes judge_score. Without the flag judge_score is never written. Exits 2 if the variable is not set.

The new file also records the sha256 and question count of the eval set. baseline has no --report-file. See Update the baseline.

VariableEffect
EVALGATE_JUDGEmodule:factory. When set, run loads a judge and adds a judge_score metric. baseline reads it only with --with-judge. Importing the module runs its import-time code with your permissions. Unset by default. See Metrics.
EVALGATE_DEBUGAny non-empty value. Errors propagate with a full traceback instead of the one-line evalgate: error: ... message. Python then exits 1, so do not use it in CI.
  • The report goes to stdout.
  • Errors go to stderr as evalgate: error: <message>.
  • baseline prints a single wrote <path> line to stdout.

--app and EVALGATE_JUDGE name Python modules that evalgate imports. Importing runs the module’s top-level code with your permissions, before any check. Only point them at code you trust.