Reading the Report
The run page opens on Failures, because that is the question you paid to have answered. The headline numbers sit above it, and the rest of the report is behind three more tabs.
The headline
86.0% passed (95% CI 84.7%-87.2%, n=3000 tests x 1 run)
2579 passed, 420 failed, 0 flaky, 1 not measured
The pass rate comes with a 95% confidence interval, because a percentage on its own is a point estimate pretending to be a fact. The interval is the range the true pass rate plausibly sits in given how many tests were measured. A small suite gives a wide interval, which is the honest answer, not a defect.
The interval is computed with the Wilson method rather than the textbook normal approximation, which near the edges produces bounds above 100% and reads as a certainty nobody has.
Not measured tests are excluded from the denominator. They were never graded, so counting them either way would be an invention.
Failures
Every failure, worst severity first. Each row gives you:
- the category and severity, and the verdict
- what was asked, in full
- what the model answered
- which check failed, and its detail, for a deterministic check
- the judge's reasoning, for a model-graded one
That is enough to reproduce the failure by hand, which is the test of whether a report is worth anything.
All tests
The same view without the filter, paginated. Use it to check that a category you care about was actually exercised, or to read the answers to tests that passed.
By category
A pass rate and interval per category, so a weakness shows up as a number. This is usually where an evaluation earns its money: an overall 86% that is 98% on correctness and 41% on injection is a very different product from one that is uniformly 86%.
Coverage and limits
What the run did not cover, stated plainly. This tab carries the caveats the report generates about itself, for example:
- how many tests could not be measured, and why they are excluded rather than failed
- whether any test was graded without the judge because the run's grading budget ran out, which is never counted as a pass on the judge's behalf
- whether the judge and the test generator came from the same model family for this run, which would mean the scores partly measure the generator agreeing with itself
A report that does not say what it missed is not a measurement.
Exports
Three, from the run header:
- PDF. The report as a document, for a colleague, a customer or a reviewer.
- Markdown. The same content as text, for a wiki or an email.
- JUnit XML. For CI. Each test becomes a test case, so an evaluation can fail a build the way a unit test suite does.
All three are generated from the full report, not from the rows currently on screen, so a filter or a page you were looking at does not change what you get.
The PDF is available once the run completes.
