Comparing Runs
Two pass rates side by side do not tell you whether a change helped. 86% against 84% could be two points of real regression or it could be noise, and it could hide forty tests that broke while forty others were fixed.
Comparing two runs answers it properly, because both runs asked the same questions.
Running a comparison
On the Runs tab, pick a baseline and a candidate, then compare. The baseline is what you had, the candidate is what you are proposing.
Runs of different suites can still be compared: tests are matched by their input rather than by a row id, so whatever the two runs have in common is paired and the rest is reported as added or removed.
What you get
- Regressed. Tests the baseline passed and the candidate failed.
- Improved. Tests the baseline failed and the candidate passed.
- No change. Passed both times or failed both times.
- Excluded as flaky. Tests that were undecided on one side. They carry no information about which model is better, so they are set aside rather than counted.
- Pass rate change, with the significance verdict in plain words above it.
Regressions come first, with the question, the baseline answer and the candidate answer in the same row.
How significance is decided
Both runs executed the same tests, so the data is paired and Codity uses McNemar's exact test. It looks only at the tests whose verdict changed, and ignores the ones that agree, because a test both models pass carries no information about which is better.
The exact binomial form is used rather than the chi-square approximation, because the number of changed tests is routinely under 25, which is exactly where the approximation stops being trustworthy.
The name of the test that ran is stored on the comparison, so a p-value can never be re-read later under the wrong assumption.
Per-category results
Each category is tested separately, and marked real change or within noise.
Asking that question of fifteen categories at the usual threshold produces a false alarm in most runs by construction, so the per-category results are corrected for the fact that many questions were asked at once. The run-level result is deliberately not corrected: it is one hypothesis, not one of many.
info"Within noise" does not mean nothing changed. It means this run does not have enough evidence to say it did. More runs per test, or more tests in that category, is what buys the power to tell.
Using it as a gate
The comparison reports whether the candidate regressed significantly, which is the check to put in front of a release. With the JUnit XML export from each run, the same thing can run in CI.
A sensible pattern is to keep one evaluation per product, run it against every model or prompt you are considering, and compare each candidate to the run you are currently shipping.
