SWE-Bench Pro User Guide

SWE-Bench Pro is a challenging benchmark evaluating LLMs/Agents on long-horizon software engineering tasks. Given a repository and an issue, the model is expected to generate a patch that fixes the described problem.

Note: Since the official Docker images are all x86 architecture, SWE-bench Pro currently only supports evaluation on x86 environments. ARM environments are not supported.

1. Feature Overview

ais_bench currently supports the following SWE-Bench Pro capabilities:

Datasets: full, mini
Tasks:
- infer: call mini-swe-agent to generate patches (model_patch)
- eval: call the SWE-bench Pro harness to run evaluation and count resolved instances
Result summary: output key metrics such as accuracy and eval_resolved_instances_num

Directory ais_bench/configs/swe_bench_pro_examples/ provides the following example configs:

mini_swe_agent_swe_bench_pro_mini.py: SWE-bench Pro Mini — commonly used for quick iterations.
mini_swe_agent_swe_bench_pro_full.py: SWE-bench Pro Full — the full test set.

2. Prerequisites

Before running, make sure the following dependencies are available:

Install mini-swe-agent (required for infer)

Note: SWE-Bench Pro official organization scaleapi has adapted mini-swe-agent. You need to download the adapted version from scaleapi’s repository.

# Clone mini-swe-agent repository
git clone https://github.com/scaleapi/mini-swe-agent.git

# Enter the project directory
cd mini-swe-agent/

# Install dependencies
pip install -e .

# Return to parent directory
cd -

Install SWE-Bench_Pro (required for infer and eval)

# Clone SWE-Bench_Pro repository
git clone https://github.com/scaleapi/SWE-bench_Pro-os.git

# Enter the project directory
cd SWE-bench_Pro-os/

# Install dependencies
pip install -r requirements.txt

# Return to parent directory
cd -

Docker is available (both infer and eval depend on containerized environments)

docker --version
docker ps

3. Minimal Configuration (Run First, Tune Later)

It is recommended to start from mini_swe_agent_swe_bench_pro_mini.py and only modify the three fields in models[0]:

model: model name (required)
url: model service endpoint (OpenAI-compatible API)
api_key: service key (use EMPTY for local services)

Example (local vLLM setup):

models = [
    dict(
        attr="local",
        abbr="swebench_pro_mini_module",
        type="LiteLLMChat",
        model="qwen3",
        api_key="EMPTY",
        url="http://127.0.0.1:8000/v1",
        batch_size=4,
        generation_kwargs=dict(),
    )
]

Dataset Path Notes

Different datasets have different loading methods:

full dataset: Supports both online loading from Hugging Face and local loading
- Online loading: keep path=""
- Local loading: set path to local parquet file or directory
mini dataset: Must be prepared locally in advance, cannot be fetched online
- Download URL: https://modelers.cn/datasets/AISBench/SWE-Bench_Pro_mini
- Recommended format: parquet
- Set path to the locally downloaded parquet file or directory

SWEBP Scripts and Docker Directory Configuration

SWE-Bench Pro evaluation must specify the following two paths. There is no default behavior:

swebp_scripts_dir: Absolute path to the run_scripts directory of the SWE-bench Pro official repository
swebp_docker_dir: Absolute path to the dockerfiles directory of the SWE-bench Pro official repository

SWEBP_SCRIPT_PATH_ABS = "{your_work_dir}/SWE-bench_Pro-os/run_scripts"  # Must be specified
SWEBP_DOCKER_PATH_ABS = "{your_work_dir}/SWE-bench_Pro-os/dockerfiles"  # Must be specified

Note: You need to clone the SWE-bench Pro official repository first: git clone https://github.com/scaleapi/SWE-bench_Pro-os.git

First-Run Recommendations

Start with the mini dataset
Use batch_size=4 (each instance creates a container; large batch_size may cause host OOM)
Keep step_limit=250 (default in examples; do not change initially)

4. Run Commands

Run the following in the repository root (config is the config file path):

ais_bench ais_bench/configs/swe_bench_pro_examples/mini_swe_agent_swe_bench_pro_mini.py

The command above runs the full pipeline (all). You can also run it step by step:

# Inference only, generate predictions
ais_bench ais_bench/configs/swe_bench_pro_examples/mini_swe_agent_swe_bench_pro_mini.py -m infer

# Evaluate based on existing predictions
ais_bench ais_bench/configs/swe_bench_pro_examples/mini_swe_agent_swe_bench_pro_mini.py -m eval

Resume from Checkpoint

Use --reuse to skip completed instances, which is useful after interruptions:

ais_bench ais_bench/configs/swe_bench_pro_examples/mini_swe_agent_swe_bench_pro_mini.py -m infer --reuse

5. How to Read Outputs

The default output directory is outputs/default/<timestamp>/. The directory structure is as follows:

Inference Outputs (predictions)

├── predictions
│   └── swebench_pro_mini_model
│       ├── swebench_pro_mini_data
│       │   ├── exit_statuses.yaml     # Exit status statistics for all instances
│       │   │
│       │   ├── instance_gravitational__teleport-xxx    # Directory for instance xxx
│       │   │   ├── instance_gravitational__teleport-xxx.config.yaml
│       │   │   ├── instance_gravitational__teleport-xxx.debug.log
│       │   │   ├── instance_gravitational__teleport-xxx.info.log
│       │   │   ├── instance_gravitational__teleport-xxx.pred
│       │   │   └── instance_gravitational__teleport-xxx.traj.json
│       │   │
│       │   └── instance_qutebrowser__qutebrowser-yyy   # Directory for instance yyy
│       │       ├── instance_qutebrowser__qutebrowser-yyy.config.yaml
│       │       ├── instance_qutebrowser__qutebrowser-yyy.debug.log
│       │       ├── instance_qutebrowser__qutebrowser-yyy.info.log
│       │       ├── instance_qutebrowser__qutebrowser-yyy.pred
│       │       └── instance_qutebrowser__qutebrowser-yyy.traj.json
│       │
│       └── swebench_pro_mini_data.json    # Final inference results

Evaluation Outputs (results)

├── results
│   ├── swebench_pro_mini_model
│   │   ├── instance_gravitational__teleport-xxx      # Directory for instance xxx
│   │   │   ├── swebench_pro_mini_data_entryscript.sh
│   │   │   ├── swebench_pro_mini_data_output.json
│   │   │   ├── swebench_pro_mini_data_patch.diff
│   │   │   ├── swebench_pro_mini_data_stderr.log
│   │   │   ├── swebench_pro_mini_data_stdout.log
│   │   │   └── workspace
│   │   │
│   │   └── instance_qutebrowser__qutebrowser-yyy      # Directory for instance yyy
│   │       ├── swebench_pro_mini_data_entryscript.sh
│   │       ├── swebench_pro_mini_data_output.json
│   │       ├── swebench_pro_mini_data_patch.diff
│   │       ├── swebench_pro_mini_data_stderr.log
│   │       ├── swebench_pro_mini_data_stdout.log
│   │       └── workspace
│   │
│   └── swebench_pro_mini_model_swebench_pro_mini_data_report.json   # Final evaluation report

Key Fields in Evaluation Report

{
  "total_instances_num": 2,   // Number of dataset instances
  "total_prediction_num": 2,  // Number of inference results
  "build_patch_instances_num": 2,   // Number of instances that successfully generated patches within the step limit
  "empty_patch_instances_num": 0,   // Number of instances with empty patches (failed to complete within step limit)
  "eval_resolved_instances_num": 1,  // Number of instances evaluated as "resolved"
  "eval_unresolved_instances_num": 1,  // Number of instances evaluated as "unresolved"
  "empty_patch_instances_ids": [],   // Instance IDs that failed to complete within step limit
  "unresolved_instances_ids": [
    "instance_gravitational__teleport-xxx"   // Instance IDs evaluated as "unresolved"
  ],
  "accuracy": 50.0   // Final evaluation accuracy
}

6. Common Issues and Troubleshooting (SWEBP Error Codes)

The following error codes come from SWEBP_CODES. You can also refer to the full FAQ:

FAQ: docs/source_en/faqs/error_codes.md

1) `SWEBP-DEPENDENCY-001`: Missing mini-swe-agent

Symptom: infer fails to start with dependency import errors
Cause: mini-swe-agent is not installed
Fix: install the adapted version from scaleapi’s repository as described in the Prerequisites section

2) `SWEBP-PARAM-001`: Empty model configuration

Symptom: prompt indicates model is not configured
Cause: models[0]['model'] is empty or only whitespace
Fix: configure model/url/api_key, and ensure model is non-empty at minimum

3) `SWEBP-PARAM-002`: Invalid dataset name

Symptom: prompt indicates dataset name is not supported
Cause: name is not in the supported set full, mini
Fix: set dataset name to one of the supported values

4) `SWEBP-DATA-001`: Dataset loading failure

Symptom: online loading fails, or local files cannot be found
Cause:
- Online mode: network or Hugging Face access issues (full dataset only)
- Local mode: path does not exist, or file format is not supported
Fix:
- For full dataset: switch to local parquet file if online loading fails
- For mini dataset: ensure it has been downloaded from https://modelers.cn/datasets/AISBench/SWE-Bench_Pro_mini

5) `SWEBP-FILE-001`: Predictions file not found

Symptom: -m eval reports missing predictions
Cause: infer was not run first, or work_dir/reuse points to a different location
Fix: run -m infer first, and ensure eval and infer use the same config/output directory

6) `SWEBP-RUNTIME-001` / `SWEBP-RUNTIME-002`: Container or harness runtime failure

Symptom: Docker image pull failure, or evaluation runtime errors
Cause: unavailable images, network issues, insufficient container runtime environment, or host OOM due to insufficient memory
Fix:
- Check docker ps first
- Verify images can be pulled from Docker Hub
- If host memory is insufficient, reduce batch_size (recommended <= 4)
- Retry with --reuse to avoid recomputing completed instances

7. Advanced Tips (Optional)

For initial debugging, use mini first, then switch to full after the pipeline is stable
To reduce empty patches, prioritize improving model capability and agent prompt templates
During evaluation, focus on empty_patch_instances_ids and unresolved_instances_ids; they are often more actionable than accuracy in early iterations
SWE-Bench Pro uses Docker images for evaluation; ensure stable network for faster image pulling
Control batch_size carefully to avoid container exit due to host memory exhaustion