Skip to content

Run it in GitHub Actions

This is the real workflow from .github/workflows/ci.yml in the evalgate repository. It runs the unit tests and then the gate, and writes the Markdown report to the job summary.

.github/workflows/ci.yml
name: ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
test-and-gate:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
persist-credentials: false
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
cache: pip
cache-dependency-path: pyproject.toml
- run: python -m pip install -e ".[dev]"
- name: Unit tests
run: python -m pytest
- name: Ship gate
run: evalgate run --report-file "$GITHUB_STEP_SUMMARY"
- name: Print report on failure
if: failure()
run: cat "$GITHUB_STEP_SUMMARY"

permissions: contents: read and persist-credentials: false give the job a read-only token that is not left in the checkout. Nothing here needs to write to the repository.

concurrency with cancel-in-progress: true cancels an older run of the same ref when you push again, so a stack of pushes does not queue a stack of gates.

timeout-minutes: 5 bounds a hung job. Locally the gate against the reference app takes under half a second, because the app is offline and the set is small. Size this for your own app.

Actions are pinned to commit SHAs, with the tag in a comment. A tag can be moved; a SHA cannot.

cache: pip keyed on pyproject.toml keeps the install fast.

python -m pytest runs the test suite first, so a broken metric function is reported as a test failure before it can confuse the gate.

evalgate run --report-file "$GITHUB_STEP_SUMMARY" is the gate. Its exit code is the verdict: 0 passes the job, 1 means a regression, 2 means bad input or an error in your app, with one line on stderr. The report is printed to stdout as usual and also written to the file GitHub renders on the run page. Because evalgate’s report is Markdown, the table appears as a table. When the gate exits 1, the job fails and the summary shows which metric dropped.

The last step runs only on failure and prints the summary file into the log, so the table is visible without opening the summary tab.

Keep the checkout, Python setup and install steps, and change the gate step to name your factory:

- name: Ship gate
run: evalgate run --app mypackage.mymodule:build --report-file "$GITHUB_STEP_SUMMARY"

Notes:

  • evalgate is not on PyPI. Install it from a checkout or a path your workflow can reach, and adapt the install line to however you vendor it.
  • Your app must be importable by the evalgate command. Installing it as a package does that, see Plug in your own app.
  • Commit baseline.json and your eval set together. The gate fails with exit code 2 if the baseline is missing, and also if the eval set differs from the sha256 recorded in the baseline, so changing questions without refreshing the baseline breaks the build on purpose.
  • Keep the app deterministic and offline in CI, or the gate will flap. If you must call a model, widen the tolerance first, see Baseline and tolerance.
  • Add secrets only to steps that need them. The reference setup needs none. Remember that --app imports and runs your module’s top-level code, so do not run it on untrusted pull request code with secrets available.

A passing run exits 0. A regression exits 1 and the summary names the metrics. Bad input, or an exception from your app, exits 2 with evalgate: error: ... in the log. See Exit codes and Read a failing report.