τ²-Bench

τ²-Bench Evaluation Benchmark Introduction

τ²-Bench (Tau Squared Bench, also written as TAU2-Bench) is an authoritative large model agent evaluation benchmark launched by Princeton University and Sierra Research in 2025. It primarily evaluates Dual-Control environment’s dialogue, tool calling, and compliance capabilities. Official benchmark repository: https://github.com/sierra-research/tau2-bench

1. Core Positioning and Background

  • Predecessor: Enhanced version of Ο„-Bench, fixing numerous code issues and adding new domains.

  • Core Innovation: Dual-Control Interaction

    • AI agents share control with users over a shared dynamic environment (not one-way instructions).

    • Both parties can call tools, modify states, and verify results, closer to real collaboration scenarios.

  • Core Objective: Evaluate agents’ comprehensive capabilities in task completion, tool usage, policy compliance, and multi-turn communication.

2. Three Evaluation Domains

Covering real customer service scenarios, each domain includes independent business policies, tool sets, databases, and task sets:

  1. Airline

    • Tasks: Flight inquiry, rescheduling, refund, seat selection, baggage, mileage redemption.

    • Tools: Ticket check, rescheduling, cancellation, payment, membership inquiry.

  2. Retail (E-commerce)

    • Tasks: Order inquiry, modification, refund, exchange, complaint, coupon.

    • Tools: Order check, return, exchange, replacement, discount.

  3. Telecom (Newly Added)

    • Tasks: Plan application, data/phone bill inquiry, network troubleshooting, suspension/activation, international roaming.

    • Tools: Account opening, suspension, bill check, troubleshooting, roaming activation.

3. Core Evaluation Mechanism (Fine-grained Assessment)

Adopts 5 automatic verification methods to avoid coarse-grained issues of single keyword matching:

  • db_check: Whether the final database state meets expectations (e.g., roaming enabled).

  • tool_call: Whether tool calling sequence, parameters, and frequency are correct.

  • policy_check: Whether business rules are violated (e.g., excessive refunds, prohibited items).

  • goal_check: Whether user core needs are met.

  • dialogue_check: Whether responses are accurate, compliant, and natural.

4. Key Evaluation Metrics

  • Pass^1: Single-round task complete success rate (primary metric).

  • Pass^k: Probability of all successes in k independent runs (stability).

  • Error decomposition:

    • Reasoning errors: Logic, tool selection, policy understanding errors.

    • Interaction errors: Communication ambiguity, information missing, user collaboration failure.

  • Domain average score: Comprehensive score across the three domains (core of leaderboard).

5. Technical Features

  • Dual-control modeling: Formulated as Dec-POMDP (Decentralized Partially Observable Markov Decision).

  • Reliable user simulator: AI simulates users, strongly coupled with environment, reducing simulation bias.

  • Automatic evaluation: No human intervention throughout, reproducible and comparable.

  • Multi-mode evaluation:

    • Standard mode: Dual-control collaboration (real scenario).

    • No-User mode: Agent full control (isolated reasoning ability).

Quick Start with τ²-Bench Evaluation in AISBench

1. Prepare Inference Services

Ensure local or cloud deployment of tested inference services following OpenAI chat/completions API specification with tool call support, and inference services for simulating users (e.g., VLLM, OpenAI, etc.).

2. Install AISBench Evaluation Tool & τ²-Bench Additional Dependencies

  1. Refer to AISBench Installation Documentation to install AISBench evaluation tool.

  2. Install τ²-Bench additional dependencies:

    # Execute in AISBench tool root directory
pip install -r requirements/datasets/tau2_dependencies.txt

Note: The commit id of the tau2-bench repository used by AISBench is c5b2d228d850c59b749b93cf32c4745d3aa53967 (version from February 2025).

3. Configure Custom Configuration File for τ²-Bench Tasks

  1. Modify necessary configurations in ais_bench/configs/agent_example/tau2_bench_task.py under AISBench tool root directory (mainly configuring information about tested inference services and user-simulating inference services)

# ......
models = [
    dict(
        abbr="openai-v1-chat",
        api_key=None, # API KEY default is an invalid string, OPENAI_API_KEY will be declared internally
        agent = None,                 # Agent implementation used, default is DEFAULT_AGENT_IMPLEMENTATION
        llm_agent = "openai/qwen3",               # Required, LLM used by agent, fill in "openai/{model name of inference service}"
        llm_args_agent = { # Parameters for agent LLM, support passing other parameters compatible with openai interface format
            "api_base": "http://localhost:2498/v1", # Required, base_url of inference service
            "temperature": 0.5
        },
    )
]

# ......

sub_tasks = ["airline", "retail", "telecom"]
for task in sub_tasks:
    datasets.append(
        dict(
            abbr=f'tau2_bench_{task}',
            args = dict(
                domain = task,                      # -d, simulation domain to run, optional values: "airline", "retail", "telecom"
                num_trials = 1,                     # Number of runs per task, default is 1
                user = None,                  # User implementation used, default is DEFAULT_USER_IMPLEMENTATION
                llm_user = "openai/qwen3",                # Required, LLM used by user, fill in "openai/{model name of inference service}"
                llm_args_user = { # Parameters for user LLM, support passing other parameters compatible with openai interface format
                    "api_base": "http://localhost:2498/v1", # Required, base_url of inference service
                    "temperature": 0.0
                },
                # ......
                max_concurrency = 5,               # Maximum concurrency for a single task, default is DEFAULT_MAX_CONCURRENCY=5
            ),
        )
    )
# ......

  • The max_concurrency in the configuration file represents the maximum concurrency for a single task (β€œairline”, β€œretail”, β€œtelecom”), default value is 5.

4. Execute τ²-Bench Tasks

  1. Execute the following command in AISBench tool root directory:

    # Execute τ²-Bench tasks
ais_bench ais_bench/configs/agent_example/tau2_bench_task.py --max-num-workers 3

    • The --max-num-workers parameter indicates the maximum task concurrency. --max-num-workers 3 means the three tasks β€œairline”, β€œretail”, β€œtelecom” will be executed in parallel.

  2. Execution process dashboard example

Base path of result&log : outputs/default/20260408_091146
Task Progress Table (Updated at: 2026-04-08 10:22:37)
Page: 1/1  Total 4 rows of data
Press Up/Down arrow to page, 'P' to PAUSE/RESUME screen refresh, 'Ctrl + C' to exit

+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| Task Name                         |   Process | Progress                                                   | Time Cost   | Status   | Log Path                                        | Extend Parameters   |
+===================================+===========+============================================================+=============+==========+=================================================+=====================+
| openai-v1-chat/tau2_bench_airline |   1856223 | [######                        ] 10/50 Running TAU2 Bench  | 0:07:13     | running  | logs/eval/openai-v1-chat/tau2_bench_airline.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_retail  |   1856224 | [######                        ] 25/114 Running TAU2 Bench | 0:11:56     | running  | logs/eval/openai-v1-chat/tau2_bench_retail.out  | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_telecom |   1856222 | [##################            ] 71/114 Running TAU2 Bench | 1:09:51     | running  | logs/eval/openai-v1-chat/tau2_bench_telecom.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+

During execution, all result files will be generated in the outputs/default/{timestamp} (e.g., outputs/default/20260408_091146) directory. During the process, you can view the detailed execution logs of the corresponding tasks in outputs/default/{timestamp}/logs/eval/openai-v1-chat/tau2_bench_{task name}.out.

  1. After task execution is complete, the following accuracy results will be printed:

| dataset | version | metric | mode | total_count | openai-v1-chat |
|----- | ----- | ----- | ----- | ----- | -----|
| tau2_bench_airline | / | pass^1 | unknown | 50 | 38.00 |
| tau2_bench_retail | / | pass^1 | unknown | 114 | 21.05 |
| tau2_bench_telecom | / | pass^1 | unknown | 114 | 33.33 |
| tau2_bench_pass^1_avg | - | naive_average | unknown | / | 30.80 |
| tau2_bench_pass^1_avg-weighted | - | weighted_average | unknown | / | 29.14 |

  • tau2_bench_avg represents the simple average score across the three domains.

  • tau2_bench_avg-weighted represents the weighted average score across the three domains (weights are the number of tasks in each domain).

  1. The structure of result files in the final outputs/default/{timestamp} directory is as follows:

outputs/default/20260408_091146
β”œβ”€β”€ configs
β”‚Β Β  └── 20260409_111604_1191827.py
β”œβ”€β”€ logs # Process logs
β”‚Β Β  └── eval
β”‚Β Β      └── openai-v1-chat
β”‚Β Β          β”œβ”€β”€ tau2_bench_airline.out # Detailed execution logs for airline evaluation
β”‚Β Β          β”œβ”€β”€ tau2_bench_retail.out # Detailed execution logs for retail evaluation
β”‚Β Β          └── tau2_bench_telecom.out # Detailed execution logs for telecom evaluation
β”œβ”€β”€ results # Final results
β”‚Β Β  └── openai-v1-chat
β”‚Β Β      β”œβ”€β”€ tau2_bench_airline # Detailed execution results for airline evaluation
β”‚Β Β      β”‚Β Β  └── tau2_run_detail.json
β”‚Β Β      β”œβ”€β”€ tau2_bench_airline.json # Accuracy results for airline tasks
β”‚Β Β      β”œβ”€β”€ tau2_bench_retail # Detailed execution results for retail tasks
β”‚Β Β      β”‚Β Β  └── tau2_run_detail.json
β”‚Β Β      β”œβ”€β”€ tau2_bench_retail.json # Accuracy results for retail tasks
β”‚Β Β      β”œβ”€β”€ tau2_bench_telecom # Detailed execution results for telecom tasks
β”‚Β Β      β”‚Β Β  └── tau2_run_detail.json
β”‚Β Β      └── tau2_bench_telecom.json # Accuracy results for telecom tasks
└── summary # Final aggregated accuracy results
    β”œβ”€β”€ summary_20260409_111604.csv
    β”œβ”€β”€ summary_20260409_111604.md
    └── summary_20260409_111604.txt

Continue Evaluation After Interruption

In cases of high concurrency, some model services may return errors during extensive multi-turn conversations, causing task failures, for example:

Base path of result&log : outputs/default/20260408_091146
Task Progress Table (Updated at: 2026-04-08 10:22:37)
Page: 1/1  Total 4 rows of data
Press Up/Down arrow to page, 'P' to PAUSE/RESUME screen refresh, 'Ctrl + C' to exit

+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| Task Name                         |   Process | Progress                                                   | Time Cost   | Status   | Log Path                                        | Extend Parameters   |
+===================================+===========+============================================================+=============+==========+=================================================+=====================+
| openai-v1-chat/tau2_bench_airline |   1856223 | [######                        ] 10/50 Running TAU2 Bench  | 0:07:13     | error    | logs/eval/openai-v1-chat/tau2_bench_airline.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_retail  |   1856224 | [######                        ] 25/114 Running TAU2 Bench | 0:11:56     | error    | logs/eval/openai-v1-chat/tau2_bench_retail.out  | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_telecom |   1856222 | [##################            ] 71/114 Running TAU2 Bench | 1:09:51     | running  | logs/eval/openai-v1-chat/tau2_bench_telecom.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+

At this point, you can manually interrupt task execution, for example by pressing Ctrl + C, and then execute the following command to continue evaluation based on the previously completed evaluation progress:

# ais_bench ais_bench/configs/agent_example/tau2_bench_task.py --max-num-workers 3 --reuse {timestamp}
ais_bench ais_bench/configs/agent_example/tau2_bench_task.py --max-num-workers 3 --reuse 20260408_091146

Multiple Executions of a Single Case (pass^k)

  1. Modify the value of the num_trials parameter in ais_bench/configs/agent_example/tau2_bench_task.py under AISBench tool root directory to the number of executions needed (default is 1)

# ......
sub_tasks = ["airline", "retail", "telecom"]
for task in sub_tasks:
    datasets.append(
        dict(
            abbr=f'tau2_bench_{task}',
            args = dict(
                domain = task,                      # -d, simulation domain to run, optional values: "airline", "retail", "telecom"
                num_trials = 5,                     # Number of runs per task, default is 1
                # ......
            ),
        )
    )
# ......

  1. After executing the ais_bench ais_bench/configs/agent_example/tau2_bench_task.py --max-num-workers 3 command, each case will be executed num_trials times, and the total number in the progress bar will also increase to num_trials times.

+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| Task Name                         |   Process | Progress                                                   | Time Cost   | Status   | Log Path                                        | Extend Parameters   |
+===================================+===========+============================================================+=============+==========+=================================================+=====================+
| openai-v1-chat/tau2_bench_airline |   1856223 | [######                        ] 30/150 Running TAU2 Bench | 0:07:13     | running  | logs/eval/openai-v1-chat/tau2_bench_airline.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_retail  |   1856224 | [######                        ] 75/342 Running TAU2 Bench | 0:11:56     | running  | logs/eval/openai-v1-chat/tau2_bench_retail.out  | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+
| openai-v1-chat/tau2_bench_telecom |   1856222 | [######                        ] 76/342 Running TAU2 Bench | 1:09:51     | running  | logs/eval/openai-v1-chat/tau2_bench_telecom.out | None                |
+-----------------------------------+-----------+------------------------------------------------------------+-------------+----------+-------------------------------------------------+---------------------+

  1. The final printed accuracy results are as follows:

| dataset | version | metric | mode | total_count | openai-v1-chat |
|----- | ----- | ----- | ----- | ----- | -----|
| tau2_bench_airline | a39421 | pass^1 | gen | 10 | 46.00 |
| tau2_bench_airline | a39421 | pass^2 | gen | 10 | 37.00 |
| tau2_bench_airline | a39421 | pass^3 | gen | 10 | 34.00 |
| tau2_bench_airline | a39421 | pass^4 | gen | 10 | 32.00 |
| tau2_bench_airline | a39421 | pass^5 | gen | 10 | 30.00 |
| tau2_bench_retail | a39421 | pass^1 | gen | 23 | 32.17 |
| tau2_bench_retail | a39421 | pass^2 | gen | 23 | 19.13 |
| tau2_bench_retail | a39421 | pass^3 | gen | 23 | 13.48 |
| tau2_bench_retail | a39421 | pass^4 | gen | 23 | 8.70 |
| tau2_bench_retail | a39421 | pass^5 | gen | 23 | 4.35 |
| tau2_bench_telecom | a39421 | pass^1 | gen | 23 | 62.61 |
| tau2_bench_telecom | a39421 | pass^2 | gen | 23 | 44.78 |
| tau2_bench_telecom | a39421 | pass^3 | gen | 23 | 36.52 |
| tau2_bench_telecom | a39421 | pass^4 | gen | 23 | 32.17 |
| tau2_bench_telecom | a39421 | pass^5 | gen | 23 | 30.43 |
| tau2_bench_pass^5_avg | - | naive_average | gen | / | 21.59 |
| tau2_bench_pass^5_avg-weighted | - | weighted_average | gen | / | 19.64 |

Using the TAU2-mini Sampled Subset

TAU2-mini is a TAU2 sampled subset provided by AISBench, using K-means clustering to sample at approximately 1/10 scale of the original dataset. It yields roughly the same evaluation scores as the original dataset, making it ideal for quick model validation and reducing evaluation costs. Dataset URL: TAU2-mini.

1. Download the TAU2-mini Dataset

Download the dataset from Modelers. After downloading and extracting, note the dataset root directory path (referred to below as <TAU2_MINI_ROOT>).

2. Replace tau2’s Original Dataset Files

Find the tau2 installation path with the following command:

pip3 show tau2 | grep "Editable project location"

This produces output similar to:

Editable project location: {tau2_root}/benchmark/src/tau2

Replace tau2’s original dataset files with the TAU2-mini files (back up {tau2_root}/src/benchmark/tau2/data/tau2/domains first):

cp -r <TAU2_MINI_ROOT>/tau2_subsets/* {tau2_root}/src/benchmark/tau2/data/tau2/domains

3. Modify the AISBench tau2-bench Configuration File

Based on the Quick Start configuration above, additionally modify the task_split_name parameter in ais_bench/configs/agent_example/tau2_bench_task.py:

# ......
for task in sub_tasks:
    datasets.append(
        dict(
            abbr=f'tau2_bench_{task}',
            args = dict(
                # ......
                task_split_name = "mini",           # Use the mini split
                # ......
            ),
        )
    )

# ......

4. Run Evaluation

Same as the standard workflow:

ais_bench ais_bench/configs/agent_example/tau2_bench_task.py --max-num-workers 3

After execution, the task counts per domain become airline 5, retail 11, and telecom 11. Example accuracy results:

| dataset | version | metric | mode | total_count | openai-v1-chat |
|----- | ----- | ----- | ----- | ----- | -----|
| tau2_bench_airline | a39421 | pass^1 | gen | 5 | 40.00 |
| tau2_bench_retail | a39421 | pass^1 | gen | 11 | 27.27 |
| tau2_bench_telecom | a39421 | pass^1 | gen | 11 | 54.55 |
| tau2_bench_pass^1_avg | - | naive_average | gen | / | 40.61 |
| tau2_bench_pass^1_avg-weighted | - | weighted_average | gen | / | 40.74 |