Will this narrow exploration？

[f233338]

Thank for your great work!
I am wondering that whether the reward based on confidence would cause models to prematurely narrow exploration,
maybe resulting in confident but short answers in some domains.
Have you explored methods to mitigate this Exploitation/Exploration trade-off to encourage deeper thought?

[Levishery]

I have the same concern. It seems that optimizing for self-certainty would push the model to become more deterministic, much like setting temperature=0 during inference.

[SunwooHong]

I also have the same concern.

[f233338]

@Levishery well, tbh, in my experiments from two months ago, the model suffered from major mode collapse. It sometimes spamming repetitive tokens (likely to cheese the self-certainty score), which caused the KL to spike and the loss to just explode.
And ended up wrecking the model's reasoning and left me with significantly shorter outputs—which kind of contradicts the paper’s finding that completion length should increase. I’m wondering if it is cuz I'm applying it to a VLM.