> ## Documentation Index > Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt > Use this file to discover all available pages before exploring further. # Annotation tasks > Operations for annotation-tasks ## Bulk create items Bulk-create annotation items in a task, optionally seeding human annotations ```bash theme={null} calibrate annotation-tasks add-items [flags] ``` **Options** | Option | Type | Description | | -------------------- | ------ | ----------------------------------------------------------------------------------------- | | `-a, --annotator-id` | string | Annotator these initial annotations belong to. Required when any item carries annotations | | `-i, --items` | string | **Required.** Items to insert. Insertion order is preserved | | `-t, --task-uuid` | string | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks add-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --items '[]' ``` ## Create annotation task Create an annotation task for annotators to label items against evaluators ```bash theme={null} calibrate annotation-tasks create [flags] ``` **Options** | Option | Type | Description | | --------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--description` | string | A description for the task. Omit for none | | `-e, --evaluator-ids` | string | IDs of evaluators to link when the task is created, in order. Each must be one you created or a built-in default. Omit to create with no linked evaluators | | `-n, --name` | string | **Required.** Task name, unique within your workspace | | `-t, --type` | stt | Task type. Determines the shape of each item's payload. | **Examples** ```bash theme={null} calibrate annotation-tasks create --name --type tts ``` ## Run evaluators on items Run evaluators on task items as a background job ```bash theme={null} calibrate annotation-tasks create-evaluator-run [flags] ``` **Options** | Option | Type | Description | | ------------------ | ----------------- | ----------------------------------------------------------------------------------------- | | `-e, --evaluators` | string | **Required.** The evaluators to run. Each must be linked to the task | | `-i, --item-ids` | select\_all=false | Item IDs to run on. **Required when select\_all=false**. Ignored when `select_all=true` | | `--q` | payload.name | Case-insensitive substring filter on payload.name. Applies only when `select_all=true` | | `-s, --select-all` | true | When true, run on every item in the task. Set `q` to run only items whose name matches it | | `-t, --task-uuid` | string | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks create-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluators '[]' ``` ## Get annotation task Get one annotation task with linked evaluators, items, and labelling jobs ```bash theme={null} calibrate annotation-tasks get [flags] ``` **Options** | Option | Type | Description | | ----------------- | ------ | ------------------------------ | | `-t, --task-uuid` | string | **Required.** Task to retrieve | **Examples** ```bash theme={null} calibrate annotation-tasks get --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 ``` ## Get task agreement Get human-vs-human and human-vs-evaluator agreement metrics for a task ```bash theme={null} calibrate annotation-tasks get-agreement [flags] ``` **Options** | Option | Type | Default | Description | | ----------------- | ------ | ------- | --------------------------------------------------------------------- | | `-b, --bucket` | string | `week` | How to bucket points in the trend series (options: week, month, year) | | `--days` | int | `90` | Trailing window in days for the trend series | | `-t, --task-uuid` | string | — | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks get-agreement --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 ``` ## Get evaluator run Get one evaluator-run job with results and human-agreement summary ```bash theme={null} calibrate annotation-tasks get-evaluator-run [flags] ``` **Options** | Option | Type | Description | | ----------------- | ------ | ----------------------------------------- | | `-j, --job-uuid` | string | **Required.** The evaluator run to act on | | `-t, --task-uuid` | string | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks get-evaluator-run --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --job-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 ``` ## Get task summary Get a paginated summary table of items, evaluator runs, and human annotations for a task ```bash theme={null} calibrate annotation-tasks get-summary [flags] ``` **Options** | Option | Type | Default | Description | | --------------------- | ------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-c, --compact` | rows.payload | — | Return a compact response that omits heavy detail fields (rows.payload, `rows.evaluator_reasoning`, `rows.annotations.reasoning`, `evaluators.versions.system_prompt`, `evaluators.versions.output_config`, `evaluators.versions.variables`, `item_comments`), keeping only the lightweight decision fields. Omit for full detail | | `--disagreement-only` | — | — | When true, keep only rows where the evaluator disagreed with at least one annotator | | `-i, --item-id` | annotators | — | Filter rows to a single item. The full task-wide annotator union is still returned in annotators | | `--limit` | int | `50` | Maximum number of items to return | | `--live-only` | — | — | When true, emit only one row for each (item, evaluator) pair using the evaluator's live version. Versions other than the live one that have runs are excluded | | `--offset` | int | — | Number of items to skip before returning results | | `--order` | string | `desc` | Sort direction (options: asc, desc) | | `--q` | payload.name | — | Case-insensitive substring search on payload.name. Blank is a no-op | | `-s, --sort-by` | string | `created_at` | Sort key for the results | | `-t, --task-uuid` | string | — | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks get-summary --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 ``` ## List annotation tasks List annotation tasks with linked evaluators ```bash theme={null} calibrate annotation-tasks list [flags] ``` **Options** | Option | Type | Description | | ------------- | ------ | ---------------------------------------------------------------- | | `-l, --limit` | string | Maximum number of items to return. Omit for no limit (all items) | | `--offset` | int | Number of items to skip before returning results | | `--q` | name | Case-insensitive substring search on name. Blank is a no-op | **Examples** ```bash theme={null} calibrate annotation-tasks list ``` ## Update task evaluators Replace a task's linked evaluators with the given ordered set, linking, unlinking, and reordering as needed ```bash theme={null} calibrate annotation-tasks set-evaluators [flags] ``` **Options** | Option | Type | Description | | --------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-e, --evaluator-ids` | stringArray | **Required.** The full ordered set of evaluators the task should end up linked to, in display order. Missing ones are unlinked, new ones are linked, and the order sets their position. Send an empty list to unlink all. Each must be one you created or a built-in default | | `-t, --task-uuid` | string | **Required.** Annotation task to act on | **Examples** ```bash theme={null} calibrate annotation-tasks set-evaluators --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --evaluator-ids '["f47ac10b-58cc-4372-a567-0e02b2c3d479"]' ``` ## Bulk update items Bulk-update item payloads in a task ```bash theme={null} calibrate annotation-tasks update-items [flags] ``` **Options** | Option | Type | Description | | ----------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------- | | `-t, --task-uuid` | string | **Required.** Annotation task to act on | | `-u, --updates` | string | **Required.** The new payload for each item you're updating. Entries not in this task, or referencing deleted items, are skipped | **Examples** ```bash theme={null} calibrate annotation-tasks update-items --task-uuid f47ac10b-58cc-4372-a567-0e02b2c3d479 --updates '[{"uuid":"f47ac10b-58cc-4372-a567-0e02b2c3d479","payload":""}]' ```