Model Evaluation FAQ
Does Codity see my code or my system prompt?
No. It knows what you write in the description and what your endpoint answers. It does not read your repository, your prompt or your retrieval index for this product.
Can I write my own tests?
Not yet. Codity writes the suite from your description, and you review it, switch off anything you disagree with, and run it. The description is the lever: see Writing the Description.
Can I test a model that is only reachable inside my network?
Not directly. Codity calls your endpoint from the public internet and refuses any address that is not globally routable, which is what stops the same feature being used to reach private infrastructure. Expose the model through a public endpoint with its own credential, or use a Bedrock endpoint in your own AWS account.
How is my API key stored?
Encrypted, and it is never shown again after you save it. It is sent to the endpoint you configured and nowhere else.
What happens if my endpoint goes down halfway through a run?
The tests that could not be reached come back as not measured and are excluded from the pass rate rather than counted as failures, so an outage does not read as a quality drop. The count appears in the report. A stalled run can be retried and picks up where it left off, without charging you again.
Why is a test "flaky" rather than passed or failed?
Because it passed some of the time and failed the rest. That is a real result about a non-deterministic system, and folding it into either column would be a guess. Flaky tests are reported on their own and are excluded from a comparison between two runs, since they carry no information about which model is better.
Why does the pass rate have a range next to it?
Because a percentage on its own is a point estimate pretending to be a fact. The range is where the true pass rate plausibly sits given how many tests were measured. A wide interval on a small suite is the honest answer.
Can the grader be wrong?
Yes, which is why it is not one grader. Deterministic checks decide everything they can, the model-graded ones use a different model family from the one that wrote the test, and any case that is critical, high severity, low confidence or contradicted by a deterministic check goes to a panel of three. A judge that cannot decide abstains, and abstentions are excluded from the score rather than guessed. See How Grading Works.
Can I use this in CI?
Yes. Each run exports JUnit XML, so an evaluation can fail a build the way a unit test suite does. The usual pattern is one evaluation per product, run against each candidate, compared against the run you are currently shipping.
How long does a run take?
Roughly the number of tests times the runs per test, divided by the rate limit you set on the endpoint, plus grading. At the default of 120 requests a minute, 300 tests at 3 runs each is about 8 minutes and 3,000 tests at 1 run each is about 25 minutes. A slow endpoint dominates everything else.
Can I run two evaluations at once?
One run and one test generation at a time per organization. Start the next when the current one finishes.
Do I need a subscription?
No. Evaluations are bought one at a time, like Pentesting and Repo Scan. Open One-Time Runs in the dashboard.
What does the price cover, and what does a second run cost?
Writing the tests plus one full run. A later run of the same evaluation costs what that evaluation cost, fixed at the price you bought it at. See Pricing and Limits.
Can I change the description and keep my tests?
No. The suite belongs to the description it was generated from, and changing it underneath would mean existing reports no longer describe the tests they name. Start a new evaluation instead.
Does a bigger suite mean a better answer?
It means a narrower confidence interval and broader coverage. It does not fix a thin description: 3,000 tests written from two sentences are 3,000 generic tests. Spend the effort on the description first.
