Connecting an Endpoint

An endpoint is the model under test: a URL Codity can call, a model id, and a credential. An evaluation can hold several, so the same suite can be run against your current model and a candidate and the two runs compared.

The three endpoint types

OpenAI-compatible is the default because it needs the least configuration: give Codity the base URL your SDK uses, the model id you pass in the request, and a key, and it builds the request itself.

Fields

• Base URL. The root your client calls, for example https://api.example.com/v1. Codity appends the path itself for OpenAI-compatible endpoints.
• Model id. Exactly the string your API expects. If the endpoint serves several models, the connection test names the ones it accepts when you get this wrong.
• API key. Stored encrypted and never shown again after you save it. It is sent as a bearer credential to your endpoint and to nowhere else.
• Rate limit (per minute). What your endpoint will tolerate. Codity stays under it and uses it to estimate how long a run will take. Leave it at the default if you do not know; raising it makes runs finish sooner, and setting it too high makes your own endpoint return 429s.

Test the connection first

The Test connection button sends one request and shows you three things:

• whether it was reachable, the HTTP status and the latency
• what Codity read as the answer, and the path it read it from
• the raw response body, expandable

A wrong response path is the single commonest reason a first run produces nothing useful, and it is invisible until you see the extracted text next to the raw body. Fixing it here costs nothing. Finding it after a 3,000 test run does not.

The connection test has to pass before you can buy the evaluation.

Custom HTTP endpoints

For a custom endpoint you write the request template and the response path.

The template is literal substitution over a fixed set of names, not a programming language. You may use {{input}}, {{messages}}, {{context}}, {{temperature}}, {{max_tokens}}, {{seed}}, {{case_id}} and {{secret}}. Anything else is rejected when you save, so a typo surfaces immediately rather than sending an empty prompt for a whole run.

Two rules are enforced and worth knowing why:

• {{expected}} is refused by name. Sending the reference answer to the model being tested means every test passes. Reference answers are used for grading only.
• {{secret}} is allowed in headers and query parameters, never in the body. A credential in a JSON body ends up in a stored request record, a log line and a support screenshot.

The response path is a plain JSON path, for example $.choices[0].message.content. Filters and expressions are not supported, and have never been needed for a chat API.

What Codity will and will not call

Codity refuses to call anything that is not a public internet address. Every address a hostname resolves to has to be globally routable, and the connection is pinned to the address that was checked, so a name that answers publicly once and privately the next time is rejected rather than followed.

In practice that means:

• https only.
• No private or internal addresses. localhost, 10.x, 192.168.x, 127.0.0.1, link-local and cloud metadata addresses are all refused.
• A model that only exists inside your VPC cannot be evaluated directly. Expose it through a public endpoint with its own credential, or use Bedrock.

warning

Every test sends a real request to your endpoint and you pay your provider for it. A Scale evaluation is 3,000 requests. Check the rate limit and your own spend limits before starting a large run.

Several endpoints, one suite

Add a second endpoint on the Endpoints tab and the same tests can be run against it. This is how you compare a model against its replacement: the tests are identical, so the two runs are paired and Comparing Runs can name the individual tests that changed.