Skip to content

Getting started

You need Python 3.12 or newer. evalgate has no runtime dependencies. The only dev dependency is pytest.

  1. Install from a checkout.

    evalgate is not published to PyPI. Install it from the repository root:

    Terminal window
    pip install -e ".[dev]"

    This puts an evalgate command on your path. Check it:

    Terminal window
    evalgate --version
    evalgate 0.1.0
  2. Run the gate.

    The repository ships a reference app, an eval set of 41 questions and a committed baseline.json. With no flags, evalgate run uses all three:

    Terminal window
    evalgate run
    ## evalgate: PASS
    | Metric | Baseline | Current | Delta | Tolerance | Status |
    | --- | ---: | ---: | ---: | ---: | :---: |
    | hit_at_k | 1.0000 | 1.0000 | +0.0000 | 0.0200 | PASS |
    | mrr | 0.9853 | 0.9853 | +0.0000 | 0.0200 | PASS |
    | citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS |
    | citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS |
    | answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS |
    | abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |
    No metric dropped more than its tolerance below the baseline.

    The exit code is 0. The paths evals/questions.jsonl and baseline.json are resolved from the current directory, so run it from the repository root.

  3. Break the app and watch it fail.

    --param passes keyword arguments to the app factory. The reference app accepts top_k, min_coverage, k1, b and tokenizer, so you can degrade it without editing code:

    Terminal window
    evalgate run --param top_k=1
    ## evalgate: FAIL
    | Metric | Baseline | Current | Delta | Tolerance | Status |
    | --- | ---: | ---: | ---: | ---: | :---: |
    | hit_at_k | 1.0000 | 0.9706 | -0.0294 | 0.0200 | FAIL |
    | mrr | 0.9853 | 0.9706 | -0.0147 | 0.0200 | PASS |
    | citation_correct | 0.9412 | 0.9412 | +0.0000 | 0.0200 | PASS |
    | citation_precision | 0.9024 | 0.9024 | +0.0000 | 0.0200 | PASS |
    | answer_contains_fact | 0.8824 | 0.8824 | +0.0000 | 0.0200 | PASS |
    | abstention_accuracy | 0.7143 | 0.7143 | +0.0000 | 0.0200 | PASS |
    Regressed beyond tolerance: hit_at_k.

    The exit code is 1. One question out of 34 lost its gold passage, and the 0.02 tolerance caught it.

PieceWhereWhat it is
Eval setevals/questions.jsonlOne JSON object per line. See Eval set format.
Baselinebaseline.jsonScores from a known-good run, committed to git. See Baseline and tolerance.
Appevalgate.refapp:buildA factory returning an object with answer(question). See Plug in your own app.