[Issue]: ROCm 7+ Performance regression on llama.cpp

[kyuz0]

Problem Description

There is a considerable performance regression when using Llama.cpp going from ROCM 6.4.4 to 7.2 or the ROCm nightly builds from TheRock.

Source: kyuz0/amd-strix-halo-toolboxes#45 (comment)

Llama.cpp with ROCm 6.4.4 is faster than when using 7.2, which is the worst performance regression (3x slower !!), and 7-nightlies (from TheRock), almost 2x slower than 6.4.4:

Examples:

model size params backend ngl n_ubatch fa test t/s
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 pp512 815.27 ± 7.37
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 tg128 72.97 ± 0.29
build: 8f91ca54e (7822)

rocm-7.2

model size params backend ngl n_ubatch fa test t/s
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 pp512 545.11 ± 6.65
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 tg128 73.21 ± 0.06
build: 8f91ca54e (7822)

rocm-6.4.4

model size params backend ngl n_ubatch fa test t/s
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 pp512 1648.22 ± 20.43
gpt-oss 20B MXFP4 MoE 11.27 GiB 20.91 B ROCm 999 2048 1 tg128 72.96 ± 0.05
build: 8f91ca54e (7822)

Full table here for many model architectures and quantizations: https://kyuz0.github.io/amd-strix-halo-toolboxes/

Operating System

Fedora 43 (6.18.3-200)

CPU

AMD Ryzen AI MAX 395+

GPU

Strix Halo gfx1151

ROCm Version

ROCm 7+

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

[ddpasa]

A suggestion to AMD folks: add a test for llama.cpp performance in your pipelines. A large number of people buying the Strix Halo buy it specifically for running llama.cpp on it. Any regressions here erode customer trust very quickly.

[tcgu-amd]

Hi @kyuz0 thank you for reaching out and bringing this to our attention. We are really sorry that you are experiencing these performance regressions and will be trying to get it fixed as soon as we can. Thanks!

[Djip007]

There is some issues from llama.cpp for that like ggml-org/llama.cpp#17917

For me, FP16/BF16 non quantized model that use rocblas have similar perf on all rocm (6.4.4 <=> 7.11..)
The regression is with (all?) quantized model (Q8/QK_N/MXFP4...) that did not use rocblas...

I had (by error) run a llama.cpp build with therock7.9 on a therock7.10 runtime and the perf was "good"...
So for me it is a build regression... what I don't know is if it is a "llvm" regression, or if we use different code path for never rocm release... It may even be a llvm/hip default build option change...

[jmmartinez]

Hello,

I narrowed down the issue to this compiler patch: llvm/llvm-project#147700

On gfx1100, over the branch amd/release/rocm-rel-7.2 I get: 1416.90 ± 14.42 t/s
While over amd/release/rocm-rel-7.2 with llvm/llvm-project#147700 reverted I get: 4378.87 ± 68.56 t/s.

After a quick look to the assembly files I see VGPR spills and the occupancy of several kernels dropping. This would explain the massive slowdown.

I'm working on reducing the test case.

CC @macurtis-amd

[ddpasa]

Hello,

I narrowed down the issue to this compiler patch: llvm/llvm-project#147700

On gfx1100, over the branch amd/release/rocm-rel-7.2 I get: 1416.90 ± 14.42 t/s While over amd/release/rocm-rel-7.2 with llvm/llvm-project#147700 reverted I get: 4378.87 ± 68.56 t/s.

After a quick look to the assembly files I see VGPR spills and the occupancy of several kernels dropping. This would explain the massive slowdown.

I'm working on reducing the test case.

CC @macurtis-amd

thanks @jmmartinez !

just to reiterate: I strongly recommend expanding your test suite to include llama.cpp, vllm and other software that constitute the primary use case for this hardware.

[kyuz0]

Hello,

I narrowed down the issue to this compiler patch: llvm/llvm-project#147700

On gfx1100, over the branch amd/release/rocm-rel-7.2 I get: 1416.90 ± 14.42 t/s While over amd/release/rocm-rel-7.2 with llvm/llvm-project#147700 reverted I get: 4378.87 ± 68.56 t/s.

After a quick look to the assembly files I see VGPR spills and the occupancy of several kernels dropping. This would explain the massive slowdown.

I'm working on reducing the test case.

CC @macurtis-amd

Thank you for looking into this, hopefully this can be fixed soon!

[jmmartinez]

Hello,
Just a minor update about the root cause of the issue.

The code is falling through a path that boosts the loop-unroll threshold for memory accesses were the underlying object is expected to be an LLVM alloca (while there is no alloca being involved). We boost the threshold when the underlying pointer is an alloca since they result in scratch memory accesses if not eliminated. Currently, the code is incorrectly boosting the threshold for memory accesses to __shared__.

When I add the missing check the performance gets back to normal. I'm working on a patch to fix this.

[mgehre-amd]

Hello, Just a minor update about the root cause of the issue.

The code is falling through a path that boosts the loop-unroll threshold for memory accesses were the underlying object is expected to be an LLVM alloca (while there is no alloca being involved). We boost the threshold when the underlying pointer is an alloca since they result in scratch memory accesses if not eliminated. Currently, the code is incorrectly boosting the threshold for memory accesses to __shared__.

When I add the missing check the performance gets back to normal. I'm working on a patch to fix this.

@jmmartinez, does this bug also affect Triton kernels?

[jmmartinez]

As a temporary workaround; it is possible to reduce the "bonus" that is given for local-memory accesses by passing -mllvm --amdgpu-unroll-threshold-local=600 (300 is the default for -O3 for any normal loop; currently the default value of --amdgpu-unroll-threshold-local is 1000).

Roughly, an unroll threshold of 1000 means something like allow for an unroll of 3.33x the size of the unroll allowed for a normal loop under -O3.

@jmmartinez, does this bug also affect Triton kernels?

Affect: yes, it's possible. But are there visible symptoms, like the massive slowdown in here? If so, I'd be glad to check.