[bug] qwen35 apply_rope_fusion

[li126com]

Problem

bridge/models/qwen_vl/modelling_qwen3_vl/rope.py#L318

assert not config.apply_rope_fusion

I have a question regarding the assert above. Are you sure that Qwen3.5 and the Qwen3-VL series cannot use apply_rope_fusion?

In practice, when I tested with Qwen3.5-35B under the THD setting, I found that after commenting out this assert, enabling apply_rope_fusion still achieves reward convergence fully aligned with the baseline, and even further reduces the train–inference mismatch.

So I would like to confirm whether this assert is actually necessary.

Minimal repro

no

Expected behavior

no

Affected area

area:model

Regression?

Yes

Environment

No response

Logs

[yaoyu-33]

Hi @li126com , I think we used to have these issues before. Might be able to remove this after mcore evolve. need to check on our side. Thanks for bringing it up.

[yaoyu-33]

Claude:

Why removing the assert and setting apply_rope_fusion=True appears to "work":

Even if you remove the assert and set apply_rope_fusion=True, the TE fused RoPE kernel is never actually invoked. There are two upstream guards that prevent it:

1. validate_rope_fusion_compatibility() blocks fusion for position_embedding_type="mrope" (which Qwen3-VL/Qwen3.5-VL uses), so provide() silently resets apply_rope_fusion=False before the model is created.

2. Qwen3VLSelfAttention calls apply_rotary_pos_emb_absolute() directly — this function always uses the unfused kernel path and never routes to TE's fused RoPE, regardless of the config flag.

So convergence matches the baseline because nothing actually changed — the fused kernel was never called.

Why fusion is not supported here:

Qwen3-VL uses per-token absolute positional frequencies with shape (seq_len, bs, 1, 2*dim), which differs from the standard mcore format (max_seqlen, 1, 1, 2*dim) that TE's fused kernels expect. Using fused kernels with this format would produce incorrect results.

Action taken: We've improved the assert message to explain these limitations clearly, so future users hitting it will understand why fusion is blocked.

[yaoyu-33]

PR with improved assert message: #3328

[yaoyu-33]

@li126com can you double check if your fusion is actually on in your training log?