Run it in GitHub Actions
This is the real workflow from .github/workflows/ci.yml in the evalgate repository. It runs the unit tests and then the gate, and writes the Markdown report to the job summary.
name: ci
on: pull_request: push: branches: [main]
permissions: contents: read
concurrency: group: ci-${{ github.ref }} cancel-in-progress: true
jobs: test-and-gate: runs-on: ubuntu-latest timeout-minutes: 5 steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 with: persist-credentials: false - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip cache-dependency-path: pyproject.toml - run: python -m pip install -e ".[dev]" - name: Unit tests run: python -m pytest - name: Ship gate run: evalgate run --report-file "$GITHUB_STEP_SUMMARY" - name: Print report on failure if: failure() run: cat "$GITHUB_STEP_SUMMARY"What each part does
Section titled “What each part does”permissions: contents: read and persist-credentials: false give the job a read-only token that is not left in the checkout. Nothing here needs to write to the repository.
concurrency with cancel-in-progress: true cancels an older run of the same ref when you push again, so a stack of pushes does not queue a stack of gates.
timeout-minutes: 5 bounds a hung job. Locally the gate against the reference app takes under half a second, because the app is offline and the set is small. Size this for your own app.
Actions are pinned to commit SHAs, with the tag in a comment. A tag can be moved; a SHA cannot.
cache: pip keyed on pyproject.toml keeps the install fast.
python -m pytest runs the test suite first, so a broken metric function is reported as a test failure before it can confuse the gate.
evalgate run --report-file "$GITHUB_STEP_SUMMARY" is the gate. Its exit code is the verdict: 0 passes the job, 1 means a regression, 2 means bad input or an error in your app, with one line on stderr. The report is printed to stdout as usual and also written to the file GitHub renders on the run page. Because evalgate’s report is Markdown, the table appears as a table. When the gate exits 1, the job fails and the summary shows which metric dropped.
The last step runs only on failure and prints the summary file into the log, so the table is visible without opening the summary tab.
Using it for your own app
Section titled “Using it for your own app”Keep the checkout, Python setup and install steps, and change the gate step to name your factory:
- name: Ship gate run: evalgate run --app mypackage.mymodule:build --report-file "$GITHUB_STEP_SUMMARY"Notes:
- evalgate is not on PyPI. Install it from a checkout or a path your workflow can reach, and adapt the install line to however you vendor it.
- Your app must be importable by the
evalgatecommand. Installing it as a package does that, see Plug in your own app. - Commit
baseline.jsonand your eval set together. The gate fails with exit code2if the baseline is missing, and also if the eval set differs from the sha256 recorded in the baseline, so changing questions without refreshing the baseline breaks the build on purpose. - Keep the app deterministic and offline in CI, or the gate will flap. If you must call a model, widen the tolerance first, see Baseline and tolerance.
- Add secrets only to steps that need them. The reference setup needs none. Remember that
--appimports and runs your module’s top-level code, so do not run it on untrusted pull request code with secrets available.
Reading the result
Section titled “Reading the result”A passing run exits 0. A regression exits 1 and the summary names the metrics. Bad input, or an exception from your app, exits 2 with evalgate: error: ... in the log. See Exit codes and Read a failing report.