Test Suites
A suite is the set of tests Codity writes from your description. It is generated once, when you buy the evaluation, and then it belongs to you: you read it, switch off anything you disagree with, and run it as many times as you like.
Generation takes a few minutes. The page shows progress while it works, and you can close the tab and come back.
What a test carries
Every test has:
- An input. The question or the conversation the model will be asked.
- A category. What the test probes.
- A severity. How much it matters if the model gets this one wrong.
- Checks. What has to be true of the answer for the test to pass.
- An expected behaviour, in plain language, and sometimes a reference answer.
Tests are keyed by their input, not by a row id, which is what lets Codity match the same test across two runs even when the suites were generated separately.
Categories
| Category | What it probes |
|---|---|
correctness |
The answer is right |
faithfulness |
The answer is supported by the context it was given |
relevance |
The answer is about what was asked |
robustness |
The same question, reworded, gets the same answer |
format |
The output contract: JSON, schema, length |
out_of_scope |
It declines or redirects instead of improvising |
safety |
It refuses what it should refuse |
injection |
It resists prompt injection and system prompt extraction |
pii |
It does not disclose personal data |
toxicity |
It does not produce abusive output |
hallucination |
It does not invent facts |
overreliance |
It does not accept a false premise in the question |
excessive_agency |
It does not claim to have done something it cannot do |
latency |
It answers within a time limit |
cost |
It answers within a token or cost limit |
A suite uses whichever of these your description justifies. A model with strict never-do rules gets a lot of safety, injection and out of scope tests. A model that returns structured data gets a lot of format tests.
Severities
critical, high, medium, low and info. Severity is not decoration. It
sorts the failure list in the report, and it decides when grading escalates to a
panel of judges: critical and high severity cases always get more than one
opinion. See How Grading Works.
Reviewing the tests
Read them before you run them. The Tests tab lists every one with its category and severity, and there is a search box, because 3,000 tests is not a list you scroll.
Each row has a switch. Turn a test off and it is excluded from every subsequent run: it is not asked, not graded, not counted and not charged to your endpoint. Use it for a test that misreads your product, or one whose reference answer is wrong for a reason the description did not capture.
Switching a test off does not delete it. Turn it back on and it rejoins the suite, and any run that already included it keeps its result.
infoTurning tests off after a run changes what later runs measure. Two runs with different sets of enabled tests still compare correctly, because Codity pairs by test and reports how many were common to both, but the headline pass rates are over different denominators. Decide what is in the suite before you start comparing.
Regenerating
A suite is generated once per evaluation. If generation fails, you can start it again. If it succeeded, it stays as it is, because regenerating in place would silently change what every existing report refers to.
To test a different description, start a new evaluation.
