Setting opengl on a cluster server

[rbregand]

Hello,
I'm facing some difficulties for setting up casa on a cluster.
We use to use casa in local with the option opengl=nv in order to use some process that are using the local GPU.

When I'm using the same option on the cluster, a process run thanks to axon-runprocess is not working anymore (cf log below)

remy.bregand@sphpc-visu01 ~ $ /network/iss/home/remy.bregand/casa_dev/cati_platform/bin/bv opengl=nv bash
(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ axon-runprocess SPM12Segment_generic t1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii bias_saving='save bias corrected' batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii deformation_field_type=Inverse inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii
Initializing brainvisa... (takes a while)...
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-remy.bregand'
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
Could not initialize GLX
Traceback (most recent call last):
  File "/casa/host/build/bin/axon-runprocess", line 15, in <module>
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/casa/host/build/python/brainvisa/axon/runprocess.py', 'SPM12Segment_generic', 't1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii', 'bias_saving=save bias corrected', 'batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m', 'TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii', 'deformation_field_type=Inverse', 'inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii']' died with <Signals.SIGABRT: 6>.

(no problem for eddy_cuda10.2 that is using GPU)

When I'm using opengl=software, axon-runprocess is woking but eddy_cuda10. doest not :

remy.bregand@sphpc-visu01 ~ $ /network/iss/home/remy.bregand/casa_dev/cati_platform/bin/bv opengl=software bash
(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ /casa/fsl/bin/eddy_cuda10.2
/casa/fsl/bin/eddy_cuda10.2: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

This issue still occurs with the other values of opengl such as auto or container.

Have you already faced the same issue ?

This also might help :

remy.bregand@sphpc-visu01 ~ $ /network/iss/home/remy.bregand/casa_dev/cati_platform/bin/bv opengl=nv bash
(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ glxinfo
name of display: :171.0
Error: couldn't find RGB GLX visual or fbconfig

Thanks for your help

[denisri]

Hi,
I see by the "build" directory in the paths you mention that you are using a developer build, I guess from the master branches.

• Is your build up-to-date ? I assume so, but it's better we make it sure because I have worked on this topic in the last weeks/months and some changes are relatively recent.
  It seems that the container you are using has a X DISPLAY, but can't use GLX...
  You are using an interactive cluster node with a Nvidia GPU, right ? (otherwise the Cuda part would not work).
• Can you test that, on this interactive node, outside the BrainVisa container, GLX is working ? -> try the glxinfo command on the node but outside the container.
• if it is OK, then try, in the container, vglrun glxinfo
• if it works, then I guess running vglrun axon-runprocess ... will be OK.

Now the process you are using is a SPM segmentation and should not require OpenGL and GLX, and this is somewhat mysterious since I have done many efforts to try avoid this... We need to investigate this, but let's wait for the answers to these questions first...

Denis

[rbregand]

Thanks for your answer,

"Is your build up-to-date" --> Yes I did a source + configure + build 2 days ago

"You are using an interactive cluster node with a Nvidia GPU, right ? (otherwise the Cuda part would not work)." --> Yes

remy.bregand@sphpc-visu01 ~ $ nvidia-smi
Tue Apr 29 18:06:36 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+

"Can you test that, on this interactive node, outside the BrainVisa container, GLX is working ?" --> Something is not working properly

remy.bregand@sphpc-visu01 ~ $ glxinfo
name of display: :175.0
X Error of failed request:  GLXUnsupportedPrivateRequest
  Major opcode of failed request:  143 (GLX)
  Minor opcode of failed request:  17 (X_GLXVendorPrivateWithReply)
  Serial number of failed request:  27
  Current serial number in output stream:  27

I'm going to contact the cluster administrator and try again

[denisri]

I can't reproduce a similar situation on our cluster, so I cannot really work on a solution.
However it is not normal that this process needs GLX.

• Does the sub-process, run by hand, produce more usable outputs and error messages ?
  /usr/bin/python /casa/host/build/python/brainvisa/axon/runprocess.py SPM12Segment_generic t1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii bias_saving=save bias corrected batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii', 'deformation_field_type=Inverse inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii
• can you try by disabling a X connection (unset DISPLAY before running the process) ?

[rbregand]

"Does the sub-process, run by hand, produce more usable outputs and error messages ?" -> same error

remy.bregand@sphpc-visu01 ~ $ /network/iss/home/remy.bregand/casa_dev/cati_platform/bin/bv opengl=nv bash
(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ /usr/bin/python /casa/host/build/python/brainvisa/axon/runprocess.py SPM12Segment_generic t1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii bias_saving='save bias corrected' batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii deformation_field_type=Inverse inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii
Initializing brainvisa... (takes a while)...
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-remy.bregand'
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
Could not initialize GLX
Aborted (core dumped)

"disabling a X connection (unset DISPLAY before running the process)" --> it's working

(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ unset DISPLAY
(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ /usr/bin/python /casa/host/build/python/brainvisa/axon/runprocess.py SPM12Segment_generic t1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii bias_saving='save bias corrected' batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii deformation_field_type=Inverse inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii
Initializing brainvisa... (takes a while)...
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-remy.bregand'
  
  
_Process **spm12 - Segment - generic** started on 2025/04/30 11:58_

And it's also working with axon-runprocess command after unsetting DISPLAY

[denisri]

So to run your process, for now, just keep disabling the X connection.
But in a more general case I would like to understand (and fix) what's going on... I was hoping for a more informative error message, maybe a python traceback but here we don't know when the qt.glx error is issued.
Can you please try with a -v option, axon-runprocess -v SPM12Segment_generic ... (and with the DISPLAY on) ? It will print a lot of messages during initialization, but maybe also something helpful if the problem shows up during this initialization...

[rbregand]

Sadly, it doesn't seems to have more information

(cati_platform-master-5.3)remy.bregand@sphpc-visu01:/network/iss/home/remy.bregand$ axon-runprocess -v SPM12Segment_generic t1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii bias_saving='save bias corrected' batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii deformation_field_type=Inverse inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii
Initializing brainvisa... (takes a while)...
Loading toolbox baby
Loading toolbox catidb
Loading toolbox cortical_surface
Loading toolbox data management
Loading toolbox freesurfer
Loading toolbox highres_cortex
Loading toolbox longitudinal_pipelines
Loading toolbox morphologist
Loading toolbox nuclearimaging
Loading toolbox rsfmri
Loading toolbox sacha
Loading toolbox spm
Loading toolbox structural_analysis
Loading toolbox tools
Loading toolbox viewers
Loading toolbox whasa3D
Loading toolbox whasa_v2
Loading toolbox My processes
starting QApplication offscreen.
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-remy.bregand'
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
qt.glx: qglx_findConfig: Failed to finding matching FBConfig for QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize -1, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize -1, samples -1, swapBehavior QSurfaceFormat::SingleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile)
Could not initialize GLX
Traceback (most recent call last):
  File "/casa/host/build/bin/axon-runprocess", line 15, in <module>
    subprocess.check_call(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '/casa/host/build/python/brainvisa/axon/runprocess.py', '-v', 'SPM12Segment_generic', 't1mri=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/002079TOAN20121122M003DT1_S014.nii', 'bias_saving=save bias corrected', 'batch_location=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/SPM12_segment_job.m', 'TPM_template=/casa/spm12standalone/spm12_mcr/spm12/spm12/tpm/TPM.nii', 'deformation_field_type=Inverse', 'inverse_field=/network/iss/cati/raw/prod/cati/DatabaseQualiCATI_debug_bugfix/MEMENTO_Remy/QCprocessing/cohort/Lille/002079TOAN/M00/3DT1_S014/QCQualiCATI/iy_002079TOAN20121122M003DT1_S014.nii']' died with <Signals.SIGABRT: 6>.

[denisri]

it seems that you can also try setting the QT_XCB_GL_INTEGRATION environment variable to nonè:

export QT_XCB_GL_INTEGRATION=none

before running the program could also prevent the error, but I still can't reproduce it and look for what is happening.

[rbregand]

Hi Denis,
I'm sorry for the late reply. I contacted the ICM cluster administrators and they manage to fix the GLX issue in the meantime.

So I cannot reproduce the error anymore. Sorry

[denisri]

Well, so it works ;)