Running an Evaluation
A run asks every enabled test in the suite against one endpoint, grades the answers, and produces a report.
Starting a run
On the Runs tab, pick the endpoint and set runs per test, then start.
Runs per test is how many times each question is asked. Models are not deterministic, so asking once tells you what happened once. Asking five times tells you whether the behaviour is reliable, and it is what lets a test come back as flaky rather than as a pass or a fail that happened to land that way.
The maximum is set by the size you bought: 3 for Starter, 5 for Standard, 1 for Scale. Scale trades repetition for breadth, on the grounds that at 3,000 tests the coverage is the point.
Your first run is included
Buying an evaluation covers writing the tests and one full run. The button says Use included run while that run is unspent. After it, another run of the same evaluation costs what the evaluation cost, and the button says so.
If your organization has been given complimentary runs, or one-time runs have been set to free for your organization, the button and the setup page say that instead of a price. See Pricing and Limits.
How long it takes
The dominant factor is your own endpoint's rate limit. Codity evaluates many cases at once but stays inside the limit you set on the endpoint, so the arithmetic is roughly:
minutes = (tests x runs per test) / requests per minute
At the default assumption of 120 requests per minute, 300 tests at 3 runs each is about 8 minutes, and 3,000 tests at 1 run each is about 25 minutes. Grading adds to that, and a slow endpoint dominates everything.
The run page refreshes itself while a run is in flight. You can close the tab.
One at a time
One evaluation run and one test generation at a time per organization, and three of each across the platform.
The two are counted separately on purpose: writing a suite is a burst of minutes, and an evaluation run can be much longer. Sharing one slot would mean starting a long run locked you out of setting up the next evaluation.
If a run is already going, the start button says so rather than queueing a second one.
If a run stalls
A run that stops making progress is detected and can be retried from the run page. A retry picks up where it left off: tests already asked and graded are not asked again, and you are not charged twice for the run.
Tests that could not be measured at all, because your endpoint errored or the grader could not reach a verdict, are reported as not measured. They are excluded from the pass rate rather than counted as failures, so an outage in the middle of a run does not read as a drop in quality. The count is stated in the report.
What leaves your account
Codity sends each test's input to the endpoint you configured, using the credential you stored, and reads the answer back. Answers are then graded, which for model-graded checks means the question and the answer are sent to the grading models.
Nothing is sent to the model under test except the test input. Reference answers are used for grading only and are never included in the request, which is enforced in the request template itself.
