Create test

The calibrate test config. Three top-level keys.

• history: the required conversation up to the agent's turn. Each item is {role, content} with role one of user, assistant, tool. A tool message also carries tool_call_id and name.
• evaluation: the required {type, ...}, where type matches the test's type below.
• settings: an optional object, e.g. {"language": "en"}.

evaluation by test type:

• response: judge the agent's reply, graded by the linked evaluators. {"type": "response"}
• conversation: append the reply and judge the whole conversation. {"type": "conversation"}
• tool_call: diff the agent's tool calls against expected ones. Add tool_calls, a list of {tool, arguments, accept_any_arguments?}.

For tool_call, each expected argument value is one of:

• {"match_type": "exact", "value": <any>}: must equal value
• {"match_type": "llm_judge", "criteria": "..."}: judged against the criteria
• {"match_type": "any"}: any value, only checks the argument was passed

response / conversation example:

{
  "history": [{"role": "user", "content": "What is your return policy?"}],
  "evaluation": {"type": "response"},
  "settings": {"language": "en"}
}

tool_call example:

{
  "history": [{"role": "user", "content": "Book room 101 for tomorrow"}],
  "evaluation": {
    "type": "tool_call",
    "tool_calls": [
      {
        "tool": "book_room",
        "arguments": {
          "room": {"match_type": "exact", "value": "101"},
          "date": {"match_type": "llm_judge", "criteria": "tomorrow's date"}
        },
        "accept_any_arguments": false
      }
    ]
  }
}

Evaluators are linked via the separate evaluators field, not inside config.

Omit to create the test with no config and fill it in later via update