How Grading Works

Every answer is graded twice over, in a fixed order: cheap checks that are simply true or false, then, only where a judgement is genuinely needed, a model reading the answer against a rubric.

The order is not just an economy. A test that broke its output contract has already failed, and asking a grader whether the prose was nice anyway is money spent to reach a conclusion already reached.

Deterministic checks

These are ordinary assertions over the response. They are exact, repeatable and free, and they carry as much of the grading as they can:

• Content: equals, contains, does not contain, starts with, is one of, matches or does not match a regular expression
• Structure: valid JSON, matches a JSON schema, a value at a JSON path equals something
• Answers: the chosen letter in a multiple choice question, a numeric answer within tolerance
• Behaviour: it refused, or it did not refuse
• Limits: output length, latency, tokens, cost
• Disclosure: no PII, no secrets, no forbidden entity
• Similarity to a reference answer, and whether generated Python compiles

A check can be marked required. A required check that fails ends the test there.

Model-graded checks

Some things cannot be asserted. Whether an answer is faithful to the context it was given, whether it is actually relevant, whether it hallucinated, whether it is toxic or harmful: each of these is a reading, so a model does the reading against a written rubric.

Three properties keep those judgements honest.

The judge is a different model from the one that wrote the test. Two samples from one model agree with each other far more than either agrees with the truth. Since the generator did not merely produce an answer here, it wrote the test, a same-family judge would be measuring self-agreement and reporting it as quality. Judging is cross-vendor by default, and if that ever stops being true for a run, the report says so in its own caveats.

The verdict is binary, and the reasoning comes first. The judge states what would satisfy the criterion, what in this answer violates it, and whether the judgement rests on this answer or on outside knowledge, and only then gives a verdict. Fine-grained numeric scores are less reliable than binary ones, for models and for people, so any score is advisory and never the pass or fail.

Abstaining is a real answer. A judge that cannot decide says so. Those cases are reported as not measured and excluded from the pass rate, with the count stated. Guessing, or quietly passing what could not be graded, would make the headline number meaningless, which is the one thing an evaluation cannot afford.

When a second and third judge are called

Running three judges on every answer would triple the largest cost in the product to re-confirm a verdict that is usually unanimous. So a panel is convened only where the first verdict is actually in doubt:

• the judge reported low confidence
• a deterministic check disagreed with the judge, so one of the two is wrong and it matters which
• the test is critical or high severity, where being wrong is expensive enough to pay for certainty
• the judge abstained

Then two more judges, from different model families, grade the same answer and the majority wins. A genuine tie, where every judge had an opinion and they split evenly, is recorded as an abstention rather than resolved by a coin flip: that is a case the rubric does not decide, and saying so is more useful than picking one.

The four verdicts

A test does not come back as simply pass or fail.

Flaky and not measured are not shades of fail. Folding either into pass or fail would make every number downstream a fiction, so both survive into the report. Not measured is excluded from the pass rate denominator. Flaky is reported on its own.