> ## Documentation Index
> Fetch the complete documentation index at: https://penseapp.vercel.app/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Speech to Text
> Evaluate Speech to Text providers via CLI
## Get started
```bash theme={null}
calibrate-agent stt
```
The interactive UI guides you through the full evaluation process:
1. **Language selection** — pick from 10+ supported Indic languages
2. **Provider selection** — choose providers (only those supporting your language are shown)
3. **Input directory** — path to the directory containing your audio files and reference transcripts
The input directory should have this structure:
```
/path/to/data/
├── stt.csv
└── audios/
├── audio_1.wav
└── audio_2.wav
```
The **stt.csv** file contains the reference transcriptions:
| id | text |
| -------- | ------------------------------- |
| audio\_1 | Hi |
| audio\_2 | Madam, my name is Geeta Shankar |
All audio files should be in WAV format. The evaluation script expects files
at `audios/.wav` where `` matches the `id` column in your CSV.
Refer to the [sample dataset](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/stt/sample_input) for a template.
4. **Output directory** — where results will be saved (defaults to `./out`)
5. **API keys** — enter the API keys for the selected providers
The evaluation runs providers in parallel (max 2 at a time), showing the transcriptions as they are generated.
## Evaluator configuration
By default, a text LLM judge — routed through [OpenRouter](https://openrouter.ai/) (set `OPENROUTER_API_KEY` in your environment) — evaluates whether each transcription matches the reference text semantically using the built-in **`semantic_match`** evaluator; expand **Default evaluator: semantic\_match** below for the exact `system_prompt` from the codebase. You can customize the judge model and add multiple evaluators by passing an optional config file with `--config`:
```bash theme={null}
calibrate-agent stt -p deepgram google -i ./data -o ./out --config config.json
```
Each evaluator's `system_prompt` is sent as the system message to its own dedicated LLM judge call (one call per evaluator, run in parallel). The user message contains the source/transcription pair.
The config file supports:
```json theme={null}
{
"evaluators": [
{
"id": "semantic-match-id",
"name": "semantic_match",
"system_prompt": "You are a highly accurate evaluator. You will be given a source text and a transcription. Mark True if the values represented by both strings match semantically.",
"judge_model": "openai/gpt-5.4-mini"
},
{
"id": "completeness-id",
"name": "completeness",
"system_prompt": "You are a highly accurate evaluator. You will be given a source text and a transcription. Mark True if all information from the source text is present in the transcription.",
"judge_model": "openai/gpt-5.4-mini"
}
]
}
```
| Key | Type | Description |
| ---------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluators` | array | List of evaluators. Each one becomes its own LLM call per row. |
| `evaluators[].id` | string | Optional unique id. Output `config.json` includes the raw `evaluators` list and an `evaluators_map` from id to name. |
| `evaluators[].name` | string | Unique evaluator name. Becomes the column name in the leaderboard. |
| `evaluators[].system_prompt` | string | Full system prompt used for this evaluator's LLM judge call. |
| `evaluators[].judge_model` | string | OpenRouter model id for this evaluator (default: `openai/gpt-5.4-mini`). Use any model in the [OpenRouter catalog](https://openrouter.ai/models). |
Each evaluator also accepts:
| Key | Type | Description |
| ----------- | ------- | ---------------------------------------------------------- |
| `type` | string | `"binary"` (default) or `"rating"` |
| `scale_min` | integer | Required when `type` is `"rating"`. Lowest allowed score. |
| `scale_max` | integer | Required when `type` is `"rating"`. Highest allowed score. |
Binary evaluators produce per-row pass/fail and a mean pass-rate. Rating evaluators produce an integer score on your scale and a mean score in the leaderboard.
When multiple evaluators are defined, each is scored independently — one LLM call per evaluator per row, all run in parallel — and appears as a separate column in the results and leaderboard.
Refer to the [sample config](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/examples/stt/config.json) for a template.
The `--config` flag is optional. When omitted, a single built-in
`semantic_match` evaluator scores semantic match between source and
transcription.
Matches [`DEFAULT_STT_EVALUATOR`](https://github.com/ARTPARK-SAHAI-ORG/calibrate/blob/main/calibrate/judges.py) in `calibrate/judges.py` when no `--config` is passed.
```text theme={null}
You are a highly accurate evaluator evaluating the transcription output of an STT model.
You will be given two strings - one is the source string used to produce an audio and the other is the transcription of that audio.
You need to evaluate if the two strings are the same.
# Important Instructions:
- Check whether the values represented by both the strings match. E.g. if one string says 1,2,3 but the other string says "one, two, three" or "one, 2, three", they should be considered the same as their underlying value is the same. However, if the actual values itself are different, e.g. for the name of a person or address or the value of any other key detail - that difference should be noted.
- Ignore differences like a word being split up into more than 1 word by spaces. Look at whether the values mean the same in both the strings.
- Minor differences in values of entities (e.g. proper nouns, numbers) matter and should be considered an error.
- If all the "values" for the strings match, mark it as True. Else, False.
```
## Output
Once all the providers have completed, it displays a leaderboard measuring key metrics along with bar charts for better visualization.
Each provider output directory includes:
* `results.csv` with one row per input audio. In addition to `id`, `gt`, `pred`, WER, evaluator scores, and evaluator reasoning, STT runs include `audio_duration_seconds`.
* `metrics.json` with aggregate `wer`, evaluator summaries, and a `cost` object. The cost object includes `provider`, `pricing_model`, `currency`, `billing_unit`, `total_seconds`, `audio_minutes`, the native per-minute rate (`cost_per_minute_currency`, denominated in `currency`), and `cost_usd`. Providers billed in a non-USD currency (e.g. Sarvam in INR) report the native-currency total `cost_in_currency` and, when the live exchange rate is reachable, the `conversion_rate` used (native-currency units per 1 USD) and `cost_usd`. If the exchange rate can't be fetched, the native-currency cost is still reported and `cost_usd` is omitted, so that provider is left out of the USD comparison rather than failing the run. Cost is a proportional estimate from bundled rates; provider invoices may differ because of per-file rounding, minimums, tiers, or regional pricing.
* `leaderboard/stt_leaderboard.xlsx` with a `cost_usd` column in the summary sheet (the one cost figure comparable across providers; per-unit rates and native-currency figures stay in each provider's `metrics.json`).
You can also view the generated transcript and metrics for each row of your dataset including the LLM judge score and reasoning.
Detailed explanation of all metrics and why using an LLM Judge is necessary
## Resources
See the full list of supported providers and their configuration options
You can also view the generated transcript and metrics for each row of your dataset including the LLM judge score and reasoning.
