GitHub Action

The Calibrate GitHub Action runs your agent tests automatically in CI. During deployment, you can choose what happens when a test fails:

gate (default) — Prevent merging the changes. This is the recommended mode for production CI pipelines to prevent deploying broken agents.
report — Report evaluation results without blocking merges. You may be aware of the mistakes but still want to merge the changes.

On pull requests, the action posts a comment with the evaluation results.

Prerequisites

Get your API key

Navigate to your Calibrate dashboard and generate an API key. See the API keys guide for detailed instructions.

Add GitHub Secret

Go to your repository Settings > Secrets and variables > Actions
Click New repository secret
Name: CALIBRATE_API_KEY
Value: Your Calibrate API key

Link tests to your agents

The agents in Calibrate you want to evaluate must have tests linked to them. By default, the action runs all the tests for every agent in your workspace.Optionally, if you want to restrict evaluation to a few specific agents, you can set that too. Gather the agent names from the Agents page (for example, checkout-bot, support-agent).

Quick start

Automatic PR checks

Create .github/workflows/calibrate.yml:

name: Calibrate automatic evaluation

on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  pull-requests: write

jobs:
  evaluate-agent:
    runs-on: ubuntu-latest
    steps:
      - name: Run Calibrate Evaluation
        uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
        with:
          api-key: ${{ secrets.CALIBRATE_API_KEY }}
          # Omit this field to run on all agents
          agents: checkout-bot, support-agent

Manual workflow dispatch

Create .github/workflows/manual-eval.yml:

name: Calibrate manual evaluation

on:
  workflow_dispatch:
    inputs:
      agents:
        description: "Agent names (comma-separated)"
        required: true
        type: string

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Run Evaluation
        uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
        with:
          api-key: ${{ secrets.CALIBRATE_API_KEY }}
          agents: ${{ inputs.agents }}

To trigger:

Navigate to Actions tab
Select Manual Evaluation
Click Run workflow
Enter your agent names and click Run workflow

Advanced configuration

Custom options

- name: Advanced Evaluation
  uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
  with:
    api-key: ${{ secrets.CALIBRATE_API_KEY }}
    agents: checkout-bot, support-agent
    # Fail the job on any test failure (default)
    mode: gate
    # Seconds between status polls
    poll-interval: 5
    # Max seconds to wait for runs to finish
    timeout: 1800

Using outputs

- name: Run Evaluation
  id: calibrate
  uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
  with:
    api-key: ${{ secrets.CALIBRATE_API_KEY }}
    agents: checkout-bot

- name: Post Results
  run: |
    echo "Total:  ${{ steps.calibrate.outputs.total }}"
    echo "Passed: ${{ steps.calibrate.outputs.passed }}"
    echo "Failed: ${{ steps.calibrate.outputs.failed }}"

Configuration reference

Inputs

Parameter	Type	Required	Default	Description
`api-key`	string	Yes	—	Calibrate API key (`sk_…`). Store as a secret.
`agents`	string	No	All agents	Agent names, comma- or newline-separated. Runs all linked tests for each. Omit to run every agent in the account linked to the API key.
`base-url`	string	No	`https://api.calibrate.artpark.ai`	Backend API URL. Override only for self-hosted.
`app-url`	string	No	`https://calibrate.artpark.ai`	Web UI base URL for view links in the report.
`mode`	string	No	`gate`	`gate` fails the job on any failure; `report` always succeeds.
`poll-interval`	integer	No	`5`	Seconds between status polls.
`timeout`	integer	No	`1800`	Max seconds to wait for all runs to finish.
`github-token`	string	No	`${{ github.token }}`	Token for PR comments. Requires `pull-requests: write`.

Outputs

Output	Type	Description
`total`	string	Total test cases across all agents evaluated
`passed`	string	Total test cases passed
`failed`	string	Total test cases failed

Environment variables

Variable	Required	Description
`CALIBRATE_API_KEY`	Yes	Your Calibrate API key

API details

The action talks to the Calibrate Public API with your API key. For each invocation it runs three steps.

Resolve agents

When agents is set, it resolves the names to UUIDs: Endpoint: POST https://api.calibrate.artpark.ai/agents/resolve Request:

{
  "names": ["checkout-bot", "support-agent"]
}

Response:

{
  "resolved": {
    "checkout-bot": "8f3c...",
    "support-agent": "a91d..."
  },
  "not_found": []
}

When agents is omitted, it lists every agent in the account instead: Endpoint: GET https://api.calibrate.artpark.ai/agents Response:

[{ "uuid": "8f3c...", "name": "checkout-bot" }]

Launch run

For each agent, it triggers all linked tests. The agent is selected by UUID in the URL path and the body is empty: Endpoint: POST https://api.calibrate.artpark.ai/agent-tests/agent/{agent_uuid}/run Request:

{}

Response:

{
  "task_id": "abc123"
}

Monitor run

It polls each task until it finishes: Endpoint: GET https://api.calibrate.artpark.ai/agent-tests/run/{task_id} Response:

{
  "status": "done",
  "total_tests": 10,
  "passed": 9,
  "failed": 1
}

Run statuses

Status	Description
`queued`	Waiting to start
`in_progress`	Running test cases
`done`	Successfully completed
`failed`	Run failed
`cancelled`	Run was cancelled

Examples

Environment-based testing

name: Multi-Environment Testing

on:
  push:
    branches: [main, staging, dev]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Set Environment
        id: env
        run: |
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "agents=checkout-bot" >> $GITHUB_OUTPUT
            echo "env=production" >> $GITHUB_OUTPUT
          elif [[ "${{ github.ref }}" == "refs/heads/staging" ]]; then
            echo "agents=staging-bot" >> $GITHUB_OUTPUT
            echo "env=staging" >> $GITHUB_OUTPUT
          else
            echo "agents=dev-bot" >> $GITHUB_OUTPUT
            echo "env=development" >> $GITHUB_OUTPUT
          fi

      - name: Evaluate
        uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
        with:
          api-key: ${{ secrets.CALIBRATE_API_KEY }}
          agents: ${{ steps.env.outputs.agents }}

Parallel agent testing

name: Multi-Agent Testing

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        agent:
          - { name: "checkout-bot" }
          - { name: "support-agent" }
          - { name: "returns-bot" }
    steps:
      - name: Test ${{ matrix.agent.name }}
        uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
        with:
          api-key: ${{ secrets.CALIBRATE_API_KEY }}
          agents: ${{ matrix.agent.name }}

Scheduled regression testing

name: Nightly Regression

on:
  schedule:
    - cron: "0 2 * * *" # 2 AM daily

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - name: Run Tests
        uses: ARTPARK-SAHAI-ORG/calibrate-github-action@v1
        with:
          api-key: ${{ secrets.CALIBRATE_API_KEY }}
          timeout: 1800

Troubleshooting

Invalid API key

authentication failed (HTTP 401) — check the api-key input

Solution: Verify CALIBRATE_API_KEY is set correctly in GitHub Secrets.

Invalid agent name

agent "my-agent": no agent with that name in this org

Solution: Confirm the agent name matches exactly and exists in your workspace.

Agent cannot run

agent my-agent: cannot run (HTTP 400): connection not verified

Solution: Open the agent in Calibrate, verify its connection, and ensure at least one test is linked.

Timeout

Solution: Increase timeout for larger test suites or check the Calibrate dashboard for run status.

Resources

API keys

Create and manage your API keys

API reference

Full REST API documentation

GitHub Action repository

Source, releases, and issue tracker

Get Started

Quickstart

Core Concepts

Reference

Prerequisites

Quick start

Automatic PR checks

Manual workflow dispatch

Advanced configuration

Custom options

Using outputs

Configuration reference

Inputs

Outputs

Environment variables

API details

Resolve agents

Launch run

Monitor run

Run statuses

Examples

Environment-based testing

Parallel agent testing

Scheduled regression testing

Troubleshooting

Invalid API key

Invalid agent name

Agent cannot run

Timeout

Resources

API keys

API reference

GitHub Action repository

​Prerequisites

​Quick start

​Automatic PR checks

​Manual workflow dispatch

​Advanced configuration

​Custom options

​Using outputs

​Configuration reference

​Inputs

​Outputs

​Environment variables

​API details

​Resolve agents

​Launch run

​Monitor run

​Run statuses

​Examples

​Environment-based testing

​Parallel agent testing

​Scheduled regression testing

​Troubleshooting

​Invalid API key

​Invalid agent name

​Agent cannot run

​Timeout

​Resources

API keys

API reference

GitHub Action repository

Prerequisites

Quick start

Automatic PR checks

Manual workflow dispatch

Advanced configuration

Custom options

Using outputs

Configuration reference

Inputs

Outputs

Environment variables

API details

Resolve agents

Launch run

Monitor run

Run statuses

Examples

Environment-based testing

Parallel agent testing

Scheduled regression testing

Troubleshooting

Invalid API key

Invalid agent name

Agent cannot run

Timeout

Resources