About Fig.7 of the paper

[ali-cogni]

I was wondering if you could provide the code (via verl) you used for the experiment shown in Fig. 7?
Thank you for your awesome and interesting work!

[XuandongZhao]

Currently, we only provide support for the open-r1 version. However, you can easily modify the code to achieve the same functionality: it's simply a matter of using either the current policy model or the base model to compute self-certainty.

[ghimiremukesh]

I also have a similar request, would it be possible to provide training script and config file for Intuitor-code? And also for generating Fig. 8? Specifically, I want to compute self-certainty during evaluation. Any guidance would be helpful!

Thank you for your time and great work indeed!

[backprop07]

Thank you for your interest!

1. We’ve added a sample training script for the code-corpus setup; please check the updated README for step-by-step instructions.

2. Computing self-certainty at inference time

   You may refer to the Self-Certainty repo for computing of self-certainty score. Below is a single example.

   a. Run LightEval (vLLM backend)

   lighteval vllm MODEL_ARGS "lighteval|<TASK>|0|0" \
       --use-chat-template \
       --output-dir OUTPUT_DIR \
       --save-details

   b. Post-process the saved results

   python confidence_list.py \
       --input_file OUTPUT_DIR/path_to_details.parquet \
       --model_dir /path/to/model \
       --output_file self_certainty.json

[ghimiremukesh]

Thanks a lot! :))