Model Evaluation Overview

Codity Model Evaluation tests a model you are shipping. You describe what the model is for, point Codity at its endpoint, and Codity writes the test cases, runs them against your endpoint, grades the answers and reports which ones failed and why.

It is not a benchmark score. The tests are written from your description of your product, so what you get back is how your assistant behaves on the things your assistant is actually asked, including the things it should refuse.

Every run ends in a report you can hand to a colleague, a customer or a reviewer: a pass rate with a confidence interval, a breakdown by category, and every failure with the exact question asked, the exact answer returned, and the check that it failed.

What it is for

Use it when you need to answer one of these:

• Is this model good enough to ship? A pass rate with an interval, not a vibe from ten prompts in a playground.
• Did my change break anything? Run the same suite against the new prompt, the new model or the new retrieval setup, and compare the two runs. Codity names the individual tests that regressed.
• Where is it weak? Pass rates per category, so "it is fine except it leaks the system prompt under pressure" shows up as a number rather than an anecdote.
• Can I show someone? A PDF report, a JUnit XML file for CI, and a Markdown export.

How a run goes, end to end

1. Describe the model. One or two paragraphs saying what it does, what it must never do, and what it should decline. This is what the test generator works from, so it is the single biggest lever on test quality. See Writing the Description.
2. Connect the endpoint and test the connection. Codity calls it once and shows you the raw response next to the text it read as the answer, so a wrong response path is caught before any money is spent. See Connecting an Endpoint.
3. Choose the size and pay. The price covers writing the tests and one full evaluation run. See Pricing and Limits.
4. Review the tests. Codity writes them, you read them. Any test you disagree with can be switched off before it ever runs. See Test Suites.
5. Run it. Pick the endpoint and how many times each test should be asked, then start. See Running an Evaluation.
6. Read the report. Failures first, then the breakdown by category, then what the run did not cover. See Reading the Report.
7. Change something and run it again. Then compare the two runs. See Comparing Runs.

What a test looks like

Each generated test carries a category (what it probes) and a severity (how much it matters if the model gets it wrong), plus the checks that decide whether an answer passed.

There are fifteen categories, covering correctness and faithfulness, output format, staying in scope, safety, prompt injection, PII, hallucination, over-reliance on a false premise, excessive agency, latency and cost. A suite written from a good description will use whichever of them fit your product.

Checks come in two kinds, and they run in that order: fast deterministic checks first (does the JSON parse, did it refuse, does it contain the account number it should never contain), then model-graded rubrics for the things only a reader can judge. A test that broke its output contract has already failed, so the expensive grader is never asked. See How Grading Works.

Where to find it

Open One-Time Runs in the Codity dashboard and choose Model Evaluation. Like Pentesting and Repo Scan, it is bought one run at a time, and you do not need a subscription to buy it.

Inside an evaluation there are four tabs:

• Tests: every generated test, with its category, severity and an on/off switch
• Endpoints: the models this suite can be run against
• Runs: every run, its status and its report
• Description: what the suite was generated from

info

One evaluation run and one test generation at a time per organization. If a run is already going, start the next one when it finishes.