"Recursion is detected during loading of "cv2" binary extensions"

[berrylizzard]

Context / environment

• Ubuntu 22.04 (specifically WSL)
• Python 3.10
• OpenCV 4.7.0-dev (built from source - read below)
• PyInstaller 5.9.0 with pyinstaller-hooks-contrib 2023.0
• GStreamer 1.0

End goal

Have a single file Ubuntu executable that runs some primitive video analytics (object detection) in real time on a GStreamer stream from an IP camera. 

What I am trying to do

I need OpenCV with support of GStreamer so I build OpenCV from source rather than pip install it. Specifically I follow procedure outlined here , i.e. run the following commands:
root@XXX:/home/user0# sudo apt update && sudo apt install -y cmake g++ wget unzip
root@XXX:/home/user0# wget -O opencv.zip https://github.com/opencv/opencv/archive/4.x.zip
root@XXX:/home/user0# unzip opencv.zip
root@XXX:/home/user0# mkdir -p build && cd build
root@XXX:/home/user0/build# cmake ../opencv-4.x
root@XXX:/home/user0/build# cmake --build .
root@XXX:/home/user0/build# make install

The file I am trying to package is my_opencv_app.py (it has an explicit import cv2 line in it). I then run the PyInstaller command:
sudo pyinstaller --onedir --hiddenimport numpy my_opencv_app.py

After what seems to be a successful build, I try to run the resulting app:
root@XXX:/home/user0/my_opencv_app# ./dist/my_opencv_app/my_opencv_app

but am getting an "ImportError: ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation."

What I am unable to resolve

I have tried to downgrade only OpenCV (every time building it from source) to 4.6.0, 4.5.5, 4.5.4 while keeping latest PyInstaller version but encountered the same error even though the vast majority of people encountering this issue resolved it by downgrading OpenCV to one those versions (see references).

Also tried to downgrade only PyInstaller with pyinstaller-hooks-contrib to 5.8.0, 5.7.0 and 2022.15, 2022.14 respectively while keeping OpenCV at the latest 4.7.0 but the error stays the same. 

I have also tried to modify the PyInstaller command to use --noupx, --onefile instead of --onedir, with and without --paths="/usr/local/lib/python3.10/dist-packages/cv2/python-3.10/" - all without success.

Note that running the original my_opencv_app.py with python works perfectly fine, the issue arises only when I try to package it with PyInstaller and run the resulting executable.

In addition, following recommendations from "Make sure everything is packaged" I looked through PYZ-00.pyz in pyi-archive_viewer (I question why I didn't see cv2 in the list but I don't know what to do to properly add it) and strace output but neither have been helpful for me, probably due to the lack of understanding of what I am looking for. The only thing that caught my attention was "WARNING: Hidden import "cv2.cv2" not found!" within the output of the PyInstaller command which is expected due to the module swap and the change in the hook for OpenCV.

References of what seems to be the same issue:

• (most recently updated - Feb 23, 2023) ImportError: ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation. opencv/opencv-python#680
• Recursion when importing cv2 from debugger opencv/opencv-python#689
• https://elinux.org/Jetson/L4T/TRT_Customized_Example#PyInstaller_with_OpenCV

The ask

• Is there a specific combination of package versions (PyInstaller, OpenCV, any others) that would solve the issue that I have not tried yet?
• Would appreciate any pointers since I have run out of ideas.

Attachments

Output of sudo pyinstaller --onedir --noupx --hiddenimport numpy --paths="/usr/local/lib/python3.10/dist-packages/cv2/python-3.10/" my_opencv_app.py command

64 INFO: PyInstaller: 5.9.0
64 INFO: Python: 3.10.6
64 INFO: Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
64 INFO: wrote /home/user0/my_opencv_app/my_opencv_app.spec
65 INFO: UPX is not available.
66 INFO: Extending PYTHONPATH with paths
['/home/user0/my_opencv_app',
 '/usr/local/lib/python3.10/dist-packages/cv2/python-3.10']
230 INFO: checking Analysis
231 INFO: Building Analysis because Analysis-00.toc is non existent
231 INFO: Initializing module dependency graph...
231 INFO: Caching module graph hooks...
236 INFO: Analyzing base_library.zip ...
535 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
756 INFO: Loading module hook 'hook-heapq.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
1371 INFO: Loading module hook 'hook-pickle.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
2212 INFO: Caching module dependency graph...
2281 INFO: running Analysis Analysis-00.toc
2289 INFO: Analyzing /home/user0/my_opencv_app/my_opencv_app.py
2291 INFO: Analyzing hidden import 'numpy'
2299 INFO: Loading module hook 'hook-numpy.py' from '/usr/local/lib/python3.10/dist-packages/numpy/_pyinstaller'...
2613 INFO: Loading module hook 'hook-difflib.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
2717 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
2788 INFO: Loading module hook 'hook-xml.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
3110 INFO: Loading module hook 'hook-platform.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
3253 INFO: Loading module hook 'hook-sysconfig.py' from '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks'...
3802 INFO: Processing module hooks...
3802 INFO: Loading module hook 'hook-cv2.py' from '/usr/local/lib/python3.10/dist-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
3803 WARNING: Hidden import "cv2.cv2" not found!
3842 INFO: Looking for ctypes DLLs
3847 INFO: Analyzing run-time hooks ...
3849 INFO: Including run-time hook '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
3850 INFO: Including run-time hook '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
3850 INFO: Including run-time hook '/usr/local/lib/python3.10/dist-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
3856 INFO: Looking for dynamic libraries
/usr/local/lib/python3.10/dist-packages/PyInstaller/building/build_main.py:177: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
  __import__(package)
286 WARNING: Cannot find libquadmath-96973f99.so.0.0.0 (needed by /usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0)
289 WARNING: Cannot find libgfortran-040039e1.so.5.0.0 (needed by /usr/local/lib/python3.10/dist-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so)
4667 INFO: Looking for eggs
4667 INFO: Python library not among binary dependencies. Performing additional search...
4678 INFO: Using Python library /lib/x86_64-linux-gnu/libpython3.10.so.1.0
4682 INFO: Warnings written to /home/user0/my_opencv_app/build/my_opencv_app/warn-my_opencv_app.txt
4700 INFO: Graph cross-reference written to /home/user0/my_opencv_app/build/my_opencv_app/xref-my_opencv_app.html
4705 INFO: Appending 'datas' from .spec
4710 INFO: checking PYZ
4710 INFO: Building PYZ because PYZ-00.toc is non existent
4710 INFO: Building PYZ (ZlibArchive) /home/user0/my_opencv_app/build/my_opencv_app/PYZ-00.pyz
5032 INFO: Building PYZ (ZlibArchive) /home/user0/my_opencv_app/build/my_opencv_app/PYZ-00.pyz completed successfully.
5035 INFO: checking PKG
5035 INFO: Building PKG because PKG-00.toc is non existent
5035 INFO: Building PKG (CArchive) my_opencv_app.pkg
5046 INFO: Building PKG (CArchive) my_opencv_app.pkg completed successfully.
5046 INFO: Bootloader /usr/local/lib/python3.10/dist-packages/PyInstaller/bootloader/Linux-64bit-intel/run
5046 INFO: checking EXE
5046 INFO: Building EXE because EXE-00.toc is non existent
5046 INFO: Building EXE from EXE-00.toc
5046 INFO: Copying bootloader EXE to /home/user0/my_opencv_app/build/my_opencv_app/my_opencv_app
5046 INFO: Appending PKG archive to custom ELF section in EXE
5051 INFO: Building EXE from EXE-00.toc completed successfully.
5052 INFO: checking COLLECT
5052 INFO: Building COLLECT because COLLECT-00.toc is non existent
5052 INFO: Building COLLECT COLLECT-00.toc
5209 INFO: Building COLLECT COLLECT-00.toc completed successfully.

Traceback

Traceback (most recent call last):
  File "my_opencv_app.py", line 7, in <module>    import cv2
  File "/home/user0/my_opencv_app/dist/my_opencv_app/cv2/__init__.py", line 181, in <module>    bootstrap()
  File "/home/user0/my_opencv_app/dist/my_opencv_app/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "importlib/__init__.py", line 126, in import_module
  File "/home/user0/my_opencv_app/dist/my_opencv_app/cv2/__init__.py", line 181, in <module>    bootstrap()
  File "/home/user0/my_opencv_app/dist/my_opencv_app/cv2/__init__.py", line 76, in bootstrap
    raise ImportError('ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.')
ImportError: ERROR: recursion is detected during loading of "cv2" binary extensions. Check OpenCV installation.
[7148] Failed to execute script 'my_opencv_app' due to unhandled exception!

[rokm]

What are the contents of config*.py files in your built-from-source OpenCV installation?
I.e., /usr/local/lib/python3.10/dist-packages/cv2/config.py and (presumably) /usr/local/lib/python3.10/dist-packages/cv2/config-3.11.py?

[rokm]

If config-3.11.py contains:

PYTHON_EXTENSIONS_PATHS = [
    os.path.join(LOADER_DIR, 'python-3.11')
] + PYTHON_EXTENSIONS_PATHS

(which it should by default, unless you override OPENCV_PYTHON_INSTALL_PATH, which you do not seem to be doing), then it is likely that the only problem is that the extension is not collected because it is located in python-3.11 subdirectory, wheras the hook (which was written for PyPI wheels) expects it to be present in the main directory (hence you get the warning about hidden import cv2.cv2 not being found).

Can you try modifying your local copy of the hook (/usr/local/lib/python3.10/dist-packages/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-cv2.py) and add the following line here:

binaries += collect_dynamic_libs('cv2', search_patterns=['cv2*.so', 'cv2*.pyd'])

[berrylizzard]

Yes, the contents of my config-3.10.py matches exactly what you have mentioned above. I have added the suggested binaries line as the last line in file /usr/local/lib/python3.10/dist-packages/_pyinstaller_hooks_contrib/hooks/stdhooks/hook-cv2.py, then ran the command
sudo pyinstaller --onedir --noupx --hiddenimport numpy my_opencv_app.py and the compiled app ./dist/my_opencv_app/my_opencv_app is working now!

Thank you very much!

[Joooshe]

The issue still occured for me even after the fix from @rokm . Here is my solution, I hope this helps someone. I'm not an expert on any of these tools or on doing builds, so please correct me if I am wrong anywhere.

This GitHub issue also discusses it https://github.com/orgs/pyinstaller/discussions/7493#discussioncomment-5315487

The solution to this issue sadly did not fix my problem.

TLDR: The problem is happening because pyinstaller doesn't know how to import opencv for certain opencv versions. The fix is to create a runtime hook that appends to sys.path the location where opencv's dynamic library is.

Long:
Opencv has special import logic. When its imported, it sets up a bunch of configuration in a python script. It does this under the cv2 module in the init.py script. So at this point the cv2 module (which for our purpose is basically just an init.py file) is in sys.modules. However, this is just for the setup and this doesn't contain the actual code that does computer vision stuff. All of this is housed in the dynamic library, so the currently running cv2 module needs to re-import the dynamic library.

It does this by deleting itself from sys.modules and then reimporting itself immediately (here). The current running cv2 module HOPES (im' sure it does more than hope, but since it broke for this use case, I am saying hope) that python's import mechanism is able to find the dynamic library before it finds the python import logic in init.py because if the init.py is found first, there will be infinite import recursion (hence our lovely error).

The cv2 python module init.py script that I am working with (which is [here](https://github.com/opencv/opencv/blob/ba483dfbb8d9c1a5b9c25fcb444e76ba9df74f09/modules/python/package/cv2/__init__.py at the time of writing), puts the dynamic library in python-3.X (the init.py script itself doesn't do this, but something else in the opencv code does) AND puts the path to python-3.X in sys.path BEFORE sys._MEIPASS (which just means that the dynamic library will be found before cv2's init.py, preventing recursion). Wow, that's great, so we shouldn't have a problem! Sadly, (at least on all the machines I tested on), this path is hard coded to the build machines path, so when you give it to another machine the dynamic library can't be found and the next thing that is found is the init.py, causing recursion. Womp womp .

Fix:
Okay, now for the fix. You may have guessed, but all we need to do is put the path to cv2/python-3.X on sys.path before sys._MEIPASS. So we can create a runtime hook to do this. Below is a runtime hook that fixed my issue:

import os
import sys

cv2_python_folder = None

# Get sys._MEIPASS (only exists in PyInstaller)
meipass = getattr(sys, "_MEIPASS", None)

if meipass and getattr(sys, "frozen", False):
    # Check if <_MEIPASS>/cv2 exists
    cv2_root = os.path.join(meipass, "cv2")
    if os.path.isdir(cv2_root):
        # Look inside for a subfolder that starts with "python-"
        for name in os.listdir(cv2_root):
            if name.startswith("python-"):
                candidate = os.path.join(cv2_root, name)
                if os.path.isdir(candidate):
                    cv2_python_folder = candidate
                    break

# If we found it, put it on sys.path right before _MEIPASS
if cv2_python_folder:
    try:
        meipass_index = sys.path.index(meipass)
    except ValueError:
        meipass_index = None

    if cv2_python_folder not in sys.path:
        if meipass_index is not None:
            # Insert *right before* sys._MEIPASS
            sys.path.insert(meipass_index, cv2_python_folder)
        else:
            # Fallback: put it at the front
            sys.path.insert(0, cv2_python_folder)

I hope this helps. I am sorry in advance for any spelling or grammar mistakes! I didn't check too hard.