Skip to content

Eval set format

An eval set is a UTF-8 text file with one JSON object per line. The default path is evals/questions.jsonl; change it with --evalset. Blank lines are skipped. There is no comment syntax.

An answerable question has gold data: the passage ids a correct answer should retrieve and cite, and a fact the answer text must contain.

eval set line
{"id": "q01", "question": "What is the maximum size of a job payload?", "gold_ids": ["p02"], "gold_fact": "256 KB"}

An unanswerable question has no gold. The app is expected to abstain, which means returning an empty answer string.

eval set line
{"id": "u01", "question": "Does Tessera offer a GraphQL API?", "answerable": false}
FieldTypeRule
idstringRequired, non-empty (whitespace-only does not count). Unique within the file.
questionstringRequired, non-empty. This is what your app receives.
answerablebooleanOptional, defaults to true. Must be a real JSON true or false.
gold_idslist of stringsOptional, defaults to []. Every entry must be a non-empty string. Required (non-empty) when answerable.
gold_factstringOptional. Non-empty if present. Required when answerable.

An unanswerable question must not carry gold_ids or gold_fact. Unknown extra keys are ignored.

gold_ids is a list because more than one passage can contain the answer. A retrieval or citation hit on any one of them counts. See Metrics.

Loading stops at the first problem and the CLI exits with code 2. Each message starts with file:line so you can jump to it:

bad.jsonl:2: invalid JSON (Expecting value)
dup.jsonl:2: duplicate question id 'q1' (first seen on line 1)

The other messages, from the loader, are:

  • <where>: expected a JSON object
  • <where>: 'id' must be a non-empty string (same for question)
  • <where>: 'answerable' must be true or false
  • <where>: 'gold_ids' must be a list of non-empty strings
  • <where>: 'gold_fact' must be a non-empty string or absent
  • <where>: an answerable question needs gold_ids and gold_fact
  • <where>: an unanswerable question must not have gold_ids or gold_fact
  • <source>: eval set is empty
  • cannot read eval set <path>: <reason>
  • Include unanswerable questions. Without them abstention_accuracy has nothing to measure and is left out of the baseline. Remember that only an empty answer counts as abstaining.
  • Write gold_fact as the short string a correct answer must contain, such as 256 KB or 30 seconds, not a full sentence. Matching is a whole-token test after normalization, and answers over 600 characters never match, see Metrics.
  • Keep the set versioned next to the app. The baseline records the sha256 and question count of the eval set, so any change to the file makes evalgate run exit 2 until you refresh the baseline in the same commit, see Update the baseline. Blank lines, line endings and outer spaces on a line do not change the hash.
  • Mind the size. With 41 questions one question moves a metric by about 0.03, which is more than the default 0.02 tolerance. See Baseline and tolerance.