Skip to content

Building torch_scatter from source leads to "Detected that PyTorch and torch_scatter were compiled with different CUDA versions." #503

@JCBrouwer

Description

@JCBrouwer

I'm building locally with PyTorch 2.10 and CUDA 12.8.

I'm sure that I built with the correct version because if I just comment out the RuntimeError in the title in my local copy of torch_scatter/__init__.py everything runs fine.

The torch.ops.torch_scatter.cuda_version() returns 11050 for some reason.

> python -c 'import torch; print(torch.__version__); import torch_scatter; print(torch.ops.torch_scatter.cuda_version())'
2.10.0+cu128
11050

Reproduction steps:

# create new env
conda create -n scatter-repro python=3.11 -y
conda activate scatter-repro

# needed on my machine due to OSError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.15' not found
# maybe this is the real root cause??
conda install -c conda-forge "libstdcxx-ng>=13" -y
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# install build dependencies
pip install numpy Cython torch==2.10 -f https://download.pytorch.org/whl/cu128

# get pyg-lib wheel
pip install pyg-lib@https://data.pyg.org/whl/nightly/torch-2.10.0%2Bcu128/torch_geometric-0.6.0.dev20260301%2Bpt210cu128-cp311-cp311-manylinux_2_28_x86_64.whl 

# actually build
pip install --verbose --no-build-isolation torch_scatter torch_sparse

# test and crash
python -c 'import torch; print(torch.__version__); import torch_scatter; print(torch.ops.torch_scatter.cuda_version())'

Results in:

2.10.0+cu128
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jcbgb/anaconda3/envs/scatter-repro/lib/python3.11/site-packages/torch_scatter/__init__.py", line 60, in <module>
    raise RuntimeError(
RuntimeError: Detected that PyTorch and torch_scatter were compiled with different CUDA versions. PyTorch has CUDA version 12.8 and torch_scatter has CUDA version 11.5. Please reinstall the torch_scatter that matches your PyTorch install.

Then:

# disable the CUDA version RuntimeError
sed -i 's/if t_major != major/if t_major != major and False/g' $CONDA_PREFIX/lib/python3.11/site-packages/torch_scatter/__init__.py

# test and succeed
python -c 'import torch; print(torch.__version__); import torch_scatter; print(torch.ops.torch_scatter.cuda_version()); x=torch_scatter.scatter_add(torch.randn(10,device="cuda"), torch.arange(10, device="cuda")); print(x.shape, x.device)'

Results in:

2.10.0+cu128
11050
torch.Size([10]) cuda:0

Is there any chance we could get nightly wheels for torch_scatter and torch_sparse?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions