How to distribute position information across the heads in Qwen2-7B?

[TaegooKang]

Thanks for your great work!
I have a question about how position information is distributed to the attention heads in Qwen2-7B.
Qwen2-7B uses group query attention, and the number of key-value heads is set to 4.
However, we have 5 types of position information in P (1 for reading order and 4 for bounding box position).
In this case, how did you handle the distribution of position information across the heads?

[mosay95]

Thanks for your attention! Regarding Qwen2, we first restore the key_states and value_states to 28 heads, and then split them based on those 28 heads. The specific implementation can be referred [class Qwen2SdpaAttention] in modeling_qwen2.py. This approach is a compromise to address the insufficiency in the number of heads, but experiments have shown that it does not have a significant negative impact on the model's performance.