Contributing Guide

Thank you for your interest in the AISBench evaluation tool! We welcome all forms of contributions, including but not limited to code, documentation, issue reports, and suggestions.

Table of Contents

Task Board

We use GitHub Issues to track tasks and issues. Before starting to contribute, we recommend:

  1. Check existing Issues to understand current work priorities

  2. For new features or major changes, it is recommended to create an Issue for discussion first

  3. For bug fixes, you can directly submit a Pull Request

License

AISBench uses the Apache License 2.0 license. By contributing code to this project, you agree that your contributions will be released under the same license.

All contributors must ensure:

  • You own or have the right to license the code you submit

  • Your contributions do not infringe any third-party intellectual property rights

  • You agree to license your contributions to the project maintainers

Development

Environment Setup

  1. Python Version Requirements: Only supports Python 3.10, 3.11, or 3.12

  2. Clone Repository:

git clone https://github.com/AISBench/benchmark.git
cd benchmark/

  1. Create Development Environment (recommended to use Conda):

conda create --name ais_bench_dev python=3.10 -y
conda activate ais_bench_dev

  1. Install Development Dependencies:

# Install core dependencies
pip3 install -e ./ --use-pep517

# Install development dependencies (if needed)
pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt

Code Structure

AISBench adopts a registry-based plugin architecture:

  • Core Plugin Modules: Located in the ais_bench/benchmark/ directory

    • models/: Model runtime classes

    • datasets/: Dataset loaders

    • openicl/icl_evaluator: ICL evaluators

    • openicl/icl_inferencer: ICL inferencers

    • calculators/: Performance metric calculators

    • registry.py: Registry definitions

  • Configuration Files: Located in the ais_bench/benchmark/configs/ directory

    • models/: Model configurations

    • datasets/: Dataset configurations

    • summarizers/: Result summarizer configurations

  • Plugin System: Supports extending functionality through entry_point mechanism, see Plugin Development Guide for details

Development Workflow

  1. Create Branch:

git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix

  1. Develop: Write code, add tests, update documentation

  2. Commit Changes:

git add .
git commit -m "feat: Add new feature description"

  1. Push and Create Pull Request

Code Checking (Linting)

Before submitting code, please ensure the code complies with the project’s code standards:

Python Code Style

  • Follow PEP 8 code style guidelines

  • Use 4 spaces for indentation

  • Line length recommended not to exceed 120 characters

  • Use meaningful variable and function names

Code Formatting Tools

It is recommended to use the following tools for code formatting:

# Use black for code formatting (if project is configured)
black ais_bench/

# Use isort for import sorting (if project is configured)
isort ais_bench/

Static Checking

Run basic Python syntax checking before submitting:

python -m py_compile your_file.py

Documentation

Documentation Structure

Project documentation is located in the docs/ directory:

  • source_zh_cn/: Chinese documentation

  • source_en/: English documentation

Documentation Writing Standards

  1. Markdown Format: Use Markdown to write documentation

  2. Chinese Documentation: Use simplified Chinese, language should be concise and clear

  3. Code Examples: Provide complete runnable code examples

  4. Link Checking: Ensure all links are valid

Updating Documentation

  • When adding new features, please synchronously update related documentation

  • When modifying APIs, please update corresponding documentation

  • When adding dataset or model configurations, please provide README.md instructions

Testing

Test Structure

Test files are located in the tests/ directory:

  • UT/: Unit tests (integrated into CI)

Running Tests

Install test-related dependencies

# Run all tests
python tests/run_tests.py

# Run all tests with multiple processes
python tests/run_tests.py -p 4 # 4 processes

# Run specific test file
pytest tests/UT/test_example.py

Writing Tests

  • Unit Tests: Test functionality of single functions or classes

  • Integration Tests: Test interactions between multiple components

  • Test Naming: Use test_ prefix to name test functions

  • Test Coverage: New features should include corresponding test cases, target coverage not less than 80%

Issues

Reporting Bugs

When reporting bugs, please provide the following information:

  1. Problem Description: Clearly describe the problem phenomenon

  2. Reproduction Steps: Provide reproducible steps

  3. Expected Behavior: Describe expected behavior

  4. Actual Behavior: Describe actual behavior

  5. Environment Information:

    • Python version

    • Operating system

    • AISBench version

    • Related dependency versions

  6. Log Information: Provide relevant error logs or stack traces

Feature Requests

When proposing feature requests, please explain:

  1. Feature Description: Describe in detail the feature you want to add

  2. Use Cases: Explain the use cases and value of this feature

  3. Possible Implementation: If you have ideas, you can propose implementation suggestions

Pull Requests (PRs) and Code Reviews

Creating Pull Requests

  1. Preparation:

    • Ensure code has passed all tests

    • Ensure code complies with code standards

    • Update related documentation

  2. PR Description:

    • Clearly describe the purpose and changes of the PR

    • If it resolves an Issue, please use the Fixes #123 format to associate

    • Provide test results or screenshots (if applicable)

  3. Code Review:

    • Keep PR size moderate for easy review

    • Respond to review comments promptly

    • Make modifications based on feedback

Review Process

  1. Automated Checks: CI checks will automatically run after PR creation

  2. Code Review: Maintainers will review the code

  3. Modification Feedback: Make modifications based on review comments

  4. Merge: After review passes, maintainers will merge the PR

DCO and Signed-off-by

AISBench uses the Developer Certificate of Origin (DCO) to ensure the legality of contributions.

How to Sign Commits

Add a Signed-off-by line to the commit message:

git commit -s -m "feat: Add new feature"

The -s parameter will automatically add the Signed-off-by line.

Signed-off-by Format

Signed-off-by: Your Name <your.email@example.com>

Meaning

By adding Signed-off-by, you confirm:

  1. The contribution is your original work, or you have the right to submit it under an open source license

  2. You agree to release the contribution under Apache License 2.0

PR Title and Classification

Title Format

Use the following format:

[type][scope]<subject>

Where:

  • [type]: Commit type, see type descriptions below

  • [scope]: Affected scope (optional), such as models, datasets, docs, etc.

  • <subject>: Brief descriptive title

Types

  • [feat]: New feature

  • [fix]: Bug fix

  • [docs]: Documentation update

  • [style]: Code format adjustment (does not affect functionality)

  • [refactor]: Code refactoring

  • [test]: Test related

  • [chore]: Build process or auxiliary tool changes

  • [perf]: Performance optimization

Examples

[feat][models]Add new model backend support
[fix][datasets]Fix dataset loading error
[docs][readme]Update installation instructions
[test][calculators]Add performance calculator tests

Code Quality

Code Review Standards

Code reviews will focus on the following aspects:

  1. Functional Correctness: Whether the code implements expected functionality

  2. Code Quality: Whether the code is clear and maintainable

  3. Test Coverage: Whether there is sufficient test coverage

  4. Documentation Completeness: Whether related documentation has been updated

  5. Performance Impact: Whether there is a negative impact on performance

  6. Backward Compatibility: Whether existing functionality has been broken

Best Practices

  1. Single Responsibility: Each function/class should do only one thing

  2. DRY Principle: Avoid duplicate code

  3. Clear Comments: Add comments for complex logic

  4. Error Handling: Properly handle exception cases

  5. Type Hints: It is recommended to use type hints to improve code readability

Adding or Changing Components

Adding New Model Backends

Refer to Supporting New Models

Adding New Datasets and Accuracy Evaluators

Refer to Supporting New Datasets and Accuracy Evaluators

Adding New Inferencers

Refer to Supporting New Inferencers

Breaking Changes

Please try to keep changes concise. For major architectural changes (>500 lines of code, excluding data/config/tests), we expect you to:

  1. Discuss in Advance: Create a discussion in GitHub Issues, explaining the reason for changes and scope of impact

  2. Backward Compatibility: Try to maintain backward compatibility, or provide migration guides

  3. Documentation Updates: Update all related documentation

  4. Test Coverage: Ensure sufficient test coverage

  5. Version Notes: Detail changes in CHANGELOG

Breaking Change Examples

  • API interface changes

  • Configuration file format changes

  • Default behavior changes

  • Removal of deprecated features

Review Expectations

As a Contributor

  • Be Patient: Reviews may take some time, please be patient

  • Respond Actively: Respond to review comments promptly and make necessary modifications

  • Stay Polite: Maintain a professional and polite attitude in discussions

  • Continuous Improvement: Treat reviews as learning opportunities and continuously improve code quality

As a Reviewer

  • Respond Promptly: Review PRs as soon as possible and provide constructive feedback

  • Friendly Communication: Provide feedback in a friendly and professional manner

  • Explain Reasons: Explain why modifications are needed to help contributors understand

  • Recognize Contributions: Recognize good contributions and encourage continued participation

Acknowledgments

Thank you to all developers who have contributed to AISBench! Your contributions make this project better.

Ways to Contribute

You can contribute in the following ways:

  • πŸ› Report Bugs

  • πŸ’‘ Propose Feature Suggestions

  • πŸ“ Improve Documentation

  • πŸ’» Submit Code

  • πŸ§ͺ Write Tests

  • πŸ” Code Review

  • πŸ“’ Promote the Project

Contributor List

All contributors will be recognized in the project. Thank you for every contributor’s efforts!

If you have any questions, please feel free to raise them in GitHub Issues.