CLI reference
evalgate has two commands, run and baseline. You can invoke it as evalgate (the console script installed by pip) or python -m evalgate. Both put the current directory on sys.path explicitly, so --app mymodule:build resolves the same way either way. See Plug in your own app.
usage: evalgate [-h] [--version] {run,baseline} ...
CI ship gate for RAG and LLM apps.
positional arguments: {run,baseline} run run the eval set and gate against the baseline baseline run the eval set and write a new baseline
options: -h, --help show this help message and exit --version show program's version number and exitevalgate --version prints evalgate 0.1.0. A command is required. Running evalgate alone prints a usage error and exits 2.
The --evalset, --baseline, --app and --param flags exist on both commands. --tolerance exists on both too, but means something different: on run it overrides every tolerance, on baseline it sets the default tolerance that is written.
evalgate run
Section titled “evalgate run”Run the eval set against the app, compare to the baseline, print the report.
usage: evalgate run [-h] [--evalset EVALSET] [--baseline BASELINE] [--app APP] [--param KEY=VALUE] [--tolerance TOLERANCE] [--report-file REPORT_FILE]
options: -h, --help show this help message and exit --evalset EVALSET --baseline BASELINE --app APP module:factory (default: reference app) --param KEY=VALUE keyword argument for the app factory, repeatable --tolerance TOLERANCE override every tolerance in the baseline --report-file REPORT_FILE also write the Markdown report here| Flag | Default | Behaviour |
|---|---|---|
--evalset PATH | evals/questions.jsonl | The eval set. Relative to the current directory. |
--baseline PATH | baseline.json | The baseline to compare against. Missing file exits 2. |
--app MODULE:FACTORY | evalgate.refapp:build | The app factory, see Plug in your own app. |
--param KEY=VALUE | none | Keyword argument passed to the factory. Repeatable. Values become int, then float, then a boolean (true or false in any case), else stay strings. nan and inf are rejected. A pair without = or with an empty key exits 2. |
--tolerance X | from the baseline | A finite float that replaces every tolerance, including per-metric values. Negative, nan or inf exits 2. A non-number is rejected by argument parsing, also exit 2. |
--report-file PATH | none | Also write the Markdown report to this path, as UTF-8. The report still goes to stdout. |
run reads k from the baseline (default 3), so hit_at_k is computed with the same k the baseline used. Before running the app it checks that the eval set matches the sha256 and question count stored in the baseline, and exits 2 if it does not.
Exit codes: 0 pass, 1 regression only, 2 bad input or an error in the app, factory, judge or files. See Exit codes.
evalgate baseline
Section titled “evalgate baseline”Run the eval set and write a new baseline.
usage: evalgate baseline [-h] [--evalset EVALSET] [--baseline BASELINE] [--app APP] [--param KEY=VALUE] [--k K] [--tolerance TOLERANCE] [--allow-regression] [--with-judge]
options: -h, --help show this help message and exit --evalset EVALSET --baseline BASELINE --app APP module:factory (default: reference app) --param KEY=VALUE keyword argument for the app factory, repeatable --k K k for hit_at_k (default: keep the existing baseline's k, else 3) --tolerance TOLERANCE default tolerance (default: keep the existing one, else 0.02) --allow-regression write the baseline even if a metric is lower than the existing one --with-judge run the EVALGATE_JUDGE judge and store judge_score--evalset, --baseline, --app and --param have the same meaning and defaults as in run. Here --baseline is the file that gets written, and the one it reads to keep the existing settings. On success it prints wrote <path> and exits 0.
| Flag | Default | Behaviour |
|---|---|---|
--k N | existing baseline’s k, else 3 | The k for hit_at_k. Must be at least 1. |
--tolerance X | existing default_tolerance, else 0.02 | The default tolerance to write. Must be finite and not negative. Per-metric tolerances in the existing file are kept as they are. |
--allow-regression | off | Without it, the command refuses (exit 2) to write a baseline in which any metric is lower than in the existing file, or is missing from this run. With it, the lower numbers are written. |
--with-judge | off | Runs the judge from EVALGATE_JUDGE and writes judge_score. Without the flag judge_score is never written. Exits 2 if the variable is not set. |
The new file also records the sha256 and question count of the eval set. baseline has no --report-file. See Update the baseline.
Environment
Section titled “Environment”| Variable | Effect |
|---|---|
EVALGATE_JUDGE | module:factory. When set, run loads a judge and adds a judge_score metric. baseline reads it only with --with-judge. Importing the module runs its import-time code with your permissions. Unset by default. See Metrics. |
EVALGATE_DEBUG | Any non-empty value. Errors propagate with a full traceback instead of the one-line evalgate: error: ... message. Python then exits 1, so do not use it in CI. |
Output streams
Section titled “Output streams”- The report goes to stdout.
- Errors go to stderr as
evalgate: error: <message>. baselineprints a singlewrote <path>line to stdout.
--app and EVALGATE_JUDGE name Python modules that evalgate imports. Importing runs the module’s top-level code with your permissions, before any check. Only point them at code you trust.