Training by both internal and external rewards.

[hossein-h]

Thanks for your interesting work.
Is it possible to train the intuitor with additional external rewards (e.g. accuracy reward)?

Have you evaluated the performance when the model is trained by both self-certainty and accuracy reward? I could not find in the paper. But, I may have missed it.

Thanks and appreciate your response.

[XuandongZhao]

Yes, combining self-certainty with a true (external) reward signal is certainly a promising approach—especially since we already have access to extensive training data in domains like math and code, where verifiable rewards are available. However, in our work, we deliberately focused on using self-certainty alone to emphasize that it can serve as an effective standalone signal, enabling reinforcement learning with purely internal feedback (RLIF).

For optimal performance in tasks such as math or coding, we do recommend incorporating an accuracy-based reward as well. As we noted in the paper, this is an important direction for future research:

Combining Reward Signals. To enable a direct comparison between self-certainty and golden-answer rewards, this paper focuses exclusively on a single reward signal. However, these signals are not mutually exclusive. Future work could explore combining them—for example, by summing both signals or alternating between them depending on the availability of golden answers. Additionally, other reward types, such as formatting rewards [Guo et al., 2025], could be integrated additively to further enhance performance. Incorporating RLIF with methods like RLHF and RLVR may also advance LLM capabilities across various dimensions.

[hossein-h]

Thank you very much for the explanation.
To activate both self-certainty and accuracy rewards in your demo config file, is it enough to set the accuracy reward weight to 1.0?
Currently, it is set to 0.0 in the config:

reward_funcs:
- accuracy 
reward_weights:
- 0.0 # Include accuracy reward for monitorying purpose, remove reward_funcs and reward_weights for open-generation RL

[backprop07]

Currently, when you set the accuracy reward to 1.0, both the self-certainty reward and the accuracy reward are normalized independently and then summed to produce the final overall advantage.

[qifen220]

Currently, when you set the accuracy reward to 1.0, both the self-certainty reward and the accuracy reward are normalized independently and then summed to produce the final overall advantage.

Thx for tips
Why not simply register the uncertainty reward as a new reward func but calculate advantages seperately, does this have any special effects?