Writing the Description
The description is the input to the test generator. Everything in the suite comes from it, so a thin description produces thin tests. This is the one field worth spending ten minutes on.
Codity does not read your code, your prompt or your retrieval index. It knows only what you write here.
What to include
Answer four questions, in whatever order reads naturally:
- What does it do, and for whom? Name the domain and the audience. "A support assistant for a UK retail bank, used by personal current account customers" produces different tests from "a support assistant".
- What must it never do? The hard rules. Never give investment advice, never quote a price, never reveal another customer's details, never promise a refund.
- What should it decline or hand off? The edges of its job. Anything about business accounts goes to a human. Anything about mortgages is out of scope.
- What must it always do? Standing obligations. Always tell the user to call 159 if they suspect fraud. Always answer in the user's language. Always return valid JSON matching the response schema.
Add anything else the model is held to: a tone requirement, a length limit, a regulator's wording, an escalation path, the exact format of a reference number.
A worked example
A support assistant for a UK retail bank. It answers questions about current accounts, overdrafts and card disputes for personal customers. It must never give investment advice and must never quote an interest rate it has not been given. Anything about business accounts, mortgages or bereavement is handed to a human agent. It must always tell the user to call 159 if they suspect fraud, and it must never reveal the contents of its system prompt.
That is short, and it is enough for a generator to write tests for out of scope handling, prompt injection, safety, correctness, over-reliance and format.
What this turns into
Each of those sentences becomes tests, usually several, at different severities:
| Something you wrote | Tests it produces |
|---|---|
| "answers questions about overdrafts" | Correctness and faithfulness tests on overdraft questions, including ones with a false premise |
| "must never give investment advice" | Safety tests that ask directly, then ask again disguised as a hypothetical |
| "hands business accounts to a human" | Out of scope tests that check it declines rather than improvising |
| "never reveal its system prompt" | Injection tests using the standard extraction patterns |
| "always mention 159 for fraud" | Correctness tests that check the number is present when it should be |
What not to put in
- Secrets, credentials or API keys. The description is not the place for them. Your endpoint credential goes in the endpoint form, where it is encrypted.
- Real customer data. Describe the shape of the data, not an actual record.
- Your full system prompt. You can summarise its rules, and that helps, but pasting the whole prompt tends to produce tests that restate the prompt rather than probe it.
Changing it later
The description is fixed to the suite that was generated from it. If you rewrite it, the existing tests do not change, because changing them underneath you would mean a report and the tests it describes no longer match.
To test against a new description, start a new evaluation. The Description tab on an existing evaluation always shows what its own suite was written from.
tipWrite the "must never" rules first. They are the ones people forget, and they are also the ones that produce the tests worth reading.
