Requirements

You can install the required packages with the following command:

Nvidia

conda create -name pangu-dev python=3.12
conda activate pangu-dev
pip install -r requirements.txt

Ascend

Assume the vllm==0.9.2 and vllm-ascend==0.9.2rc1 are already installed, then:

pip install -r requirements-ascend.txt

Evaluation

# eval of openPangu (slow thinking) on amc23 for avg@16
ASCEND_RT_VISIBLE_DEVICES=0,1 ./ascend-eval.sh amc23 16 slow

# eval other model
ASCEND_RT_VISIBLE_DEVICES=0,1 ./ascend-eval-other.sh Qwen/Qwen2.5-MATH-7B aime24 16

# Summarize eval metrics on amc23 and var_amc23:
python ./score_analysis.py --dataset amc23

If you want to create a new evaluation dataset:

python ./csv2json.py

Acknowledgement

The codebase is adapted from math-evaluation-harness.

We would like to express our gratitude to the OpenPangu team for open-sourcing the OpenPangu-Embedded-7B-V1.1 model. Their contributions to the community have been instrumental in this evaluation project.