Pricing and Limits
Model Evaluation is bought one evaluation at a time. There is no plan to be on and no seat to buy.
The three sizes
| Starter | Standard | Scale | |
|---|---|---|---|
| Price | $39 | $129 | $189 |
| Tests | up to 300 | up to 1,500 | up to 3,000 |
| Runs per test | 3 | 5 | 1 |
| Good for | A first look at a model's behaviour and a failure report you can act on | A per-test pass rate with a usable confidence interval | Broad purpose and safety coverage at production scale |
The size is chosen for you from the number of tests you ask for, so type the number you want and the matching size is selected.
Runs per test is where the three differ in kind, not just in volume. Five runs is where a per-test pass rate starts to carry a usable interval and where flaky behaviour separates itself from a pass. Scale spends that budget on breadth instead: at 3,000 tests, coverage is the point.
What the price covers
- Writing the test suite from your description
- One full evaluation run, including grading
- The report, and every export of it
You are charged once, at the start, and only after the connection test has passed, so nothing is taken for an evaluation that could not have run.
Running it again
Another run of the same evaluation costs what that evaluation cost. The button on the Runs tab shows the amount.
The price is fixed to the evaluation at the moment you bought it. If the list price changes later, your existing evaluations keep the price they were bought at, in both directions.
Free and complimentary runs
Your organization may have been given either of two things by a Codity administrator:
- One-time runs set to free. Every evaluation costs nothing. The setup page shows "Free" with the standard price struck through, and the button says Generate test cases rather than offering a payment.
- Complimentary runs. A fixed number of free evaluations. The setup page says how many are left. When they are used up, the standard price applies again.
Either way the page tells you before you commit, and no payment sheet opens for something that is free.
Concurrency
| Limit | Per organization | Across the platform |
|---|---|---|
| Evaluation runs at once | 1 | 3 |
| Test generations at once | 1 | 3 |
The two are counted separately, because generating a suite takes minutes and a run can take much longer. Sharing one slot would mean a long run locked you out of setting up your next evaluation.
If you are at the limit, the page says so rather than queueing.
Your own model costs
The price above is what Codity charges. Every test is also a real request to your endpoint, billed by whoever hosts it.
A Scale evaluation is 3,000 requests. A Standard evaluation is 1,500 tests at 5 runs each, which is 7,500 requests. Check your provider's pricing and your own spend limits before starting a large run.
infoThe rate limit you set on the endpoint is the throttle. Codity stays under it, so it also controls how fast you spend at your provider.
