Create evaluator
Create an evaluator along with its first version, which is set livecalibrate evaluators create [flags]
| Option | Type | Description |
|---|---|---|
--data-type | text | The modality the judge reads: |
--description | string | Description. Omit to leave blank |
-e, --evaluator-type | tts | What the evaluator judges: |
-n, --name | string | Required. Evaluator name, unique within your workspace |
--output-type | binary | How the evaluator scores: |
-v, --version-param | string | Required. One version of an evaluator: its judge prompt, model, variables, and rubric. |
calibrate evaluators create --name <value> --version-param '{"judge_model":"<value>","system_prompt":"<value>"}'
Create evaluator version
Add a new version to an evaluator you createdcalibrate evaluators create-version [flags]
| Option | Type | Description |
|---|---|---|
-e, --evaluator-uuid | string | Required. Evaluator to add a version to |
-j, --judge-model | openai/gpt-4.1 | Required. The model that runs the judge, named the way its provider does, for example openai/gpt-4.1 or anthropic/claude-sonnet-4 |
-m, --make-live | true | When true, immediately point the evaluator’s live version at this new version |
--output-config | rating | The scale points and their labels. Required for a rating evaluator. A binary evaluator uses the default Correct/Wrong labels unless you set your own |
-s, --system-prompt | — | Required. { {variable} } Judge system prompt. May contain { {variable} } placeholders |
-v, --variables | string | Declared prompt variables. Omit if the prompt has none. After the first version the variable names are fixed. You can change a variable’s description or default, but not add, remove, or rename one |
calibrate evaluators create-version --evaluator-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --judge-model <value> --system-prompt <value>
Get evaluator
Get one evaluator with its full version historycalibrate evaluators get [flags]
| Option | Type | Description |
|---|---|---|
-c, --compact | versions.system_prompt | Return a compact response that omits heavy detail fields (versions.system_prompt, versions.output_config, versions.variables), keeping only the lightweight decision fields. Omit for full detail |
-e, --evaluator-uuid | string | Required. Evaluator to retrieve |
calibrate evaluators get --evaluator-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479
List evaluators
List your evaluatorscalibrate evaluators list [flags]
| Option | Type | Default | Description |
|---|---|---|---|
--data-type | string | — | Filter by modality. Omit for all (options: text, audio) |
-e, --evaluator-type | string | — | Filter by what the evaluator judges. Omit for all types (options: tts, stt, llm, llm-general, conversation) |
-i, --include-defaults | — | true | Retained for backward compatibility and no longer filters. Your evaluators, including your editable copies of the defaults, are always returned |
-l, --limit | string | — | Maximum number of items to return. Omit for no limit (all items) |
--offset | int | — | Number of items to skip before returning results |
--q | name | — | Case-insensitive substring search on name. Blank is a no-op |
calibrate evaluators list