> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Create test

> Create a test that runs your agent against a conversation and evaluates its answer quality or the tools it calls



## OpenAPI

````yaml /api-reference/openapi.json post /tests
openapi: 3.1.0
info:
  title: Calibrate Public API
  version: 0.1.0
  description: Programmatic API for CI/automation. Pass your key in the `X-API-Key` header.
servers:
  - url: https://api.calibrate.artpark.ai
    description: Production
security: []
paths:
  /tests:
    post:
      tags:
        - tests
      summary: Create test
      description: >-
        Create a test that runs your agent against a conversation and evaluates
        its answer quality or the tools it calls
      operationId: create_test_endpoint_tests_post
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TestCreate'
      responses:
        '200':
          description: Successful Response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TestCreateResponse'
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
      security:
        - ApiKeyAuth: []
components:
  schemas:
    TestCreate:
      properties:
        name:
          type: string
          title: Name
          description: Name of the test, unique within the workspace
        type:
          type: string
          enum:
            - response
            - tool_call
            - conversation
          title: Type
          description: |
            What the test judges:

            - `response`: judges the generated reply
            - `tool_call`: diffs the generated tool calls
            - `conversation`: judges the full conversation
        config:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          description: >-
            The calibrate test config. Three top-level keys.


            - `history`: the required conversation up to the agent's turn. Each
            item is `{role, content}` with `role` one of `user`, `assistant`,
            `tool`. A `tool` message also carries `tool_call_id` and `name`.

            - `evaluation`: the required `{type, ...}`, where `type` matches the
            test's `type` below.

            - `settings`: an optional object, e.g. `{"language": "en"}`.


            `evaluation` by test type:

            - `response`: judge the agent's reply, graded by the linked
            evaluators. `{"type": "response"}`

            - `conversation`: append the reply and judge the whole conversation.
            `{"type": "conversation"}`

            - `tool_call`: diff the agent's tool calls against expected ones.
            Add `tool_calls`, a list of `{tool, arguments,
            accept_any_arguments?}`.


            For `tool_call`, each expected argument value is one of:

            - `{"match_type": "exact", "value": <any>}`: must equal `value`

            - `{"match_type": "llm_judge", "criteria": "..."}`: judged against
            the criteria

            - `{"match_type": "any"}`: any value, only checks the argument was
            passed


            `response` / `conversation` example:

            ```json

            {
              "history": [{"role": "user", "content": "What is your return policy?"}],
              "evaluation": {"type": "response"},
              "settings": {"language": "en"}
            }

            ```


            `tool_call` example:

            ```json

            {
              "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
              "evaluation": {
                "type": "tool_call",
                "tool_calls": [
                  {
                    "tool": "book_room",
                    "arguments": {
                      "room": {"match_type": "exact", "value": "101"},
                      "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
                    },
                    "accept_any_arguments": false
                  }
                ]
              }
            }

            ```


            Evaluators are linked via the separate `evaluators` field, not
            inside `config`.


            Omit to create the test with no config and fill it in later via
            update
          examples:
            - evaluation:
                type: response
              history:
                - content: What is your return policy?
                  role: user
              settings:
                language: en
        evaluators:
          anyOf:
            - items:
                $ref: '#/components/schemas/routers__tests__EvaluatorRef'
              type: array
            - type: 'null'
          title: Evaluators
          description: Evaluators to link. Used by `response` and `conversation` tests
      type: object
      required:
        - name
        - type
      title: TestCreate
    TestCreateResponse:
      properties:
        uuid:
          type: string
          maxLength: 36
          minLength: 36
          title: Uuid
          description: ID of the newly created test
          examples:
            - b1c2d3e4-f5a6-7890-bcde-f12345678901
        message:
          type: string
          title: Message
          description: Confirmation message
      type: object
      required:
        - uuid
        - message
      title: TestCreateResponse
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    routers__tests__EvaluatorRef:
      properties:
        evaluator_uuid:
          type: string
          maxLength: 36
          minLength: 36
          title: Evaluator Uuid
          description: Evaluator to attach to the test
          examples:
            - f47ac10b-58cc-4372-a567-0e02b2c3d479
        variable_values:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          description: >-
            Values for the evaluator's `{{placeholder}}` variables, pinned on
            this test. Omit to inherit the evaluator version's defaults
          examples:
            - criteria: The reply must cite the refund window
      additionalProperties: false
      type: object
      required:
        - evaluator_uuid
      title: EvaluatorRef
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-Key
      description: API key for authentication

````