multiple on-disk PKG files for the bundled app's start up sequence?

[KardoPaska]

Hello Friends!

Is it easily possible to implement the use of multiple on-disk PKG files for the bundled app's start up sequence (boot strap / load)?

The EXE class's keyword argument append_pkg=False hints to me that the machinery might already be there.

But I can't figure out how to hack a pointer from the bundled PKG file to point to another on-disk PKG file so it can be used during the boot. Or vice-versa, having the satellite PKG file point to the bundled PKG file for use.

With onedir mode, i can re-use the _internal folder contents and just send new exe files as an update. But the exe file is still >30 MB, too large for an email attachment.

And looking at the PYZ-00.toc file, it seems a lot of the contents contributing to the bundled app (.exe) size are located in Lib\site-packages, like numpy, pandas, scipy, matplotlib, etc...

Can we put these into a separate PKG file for the exe? Or bundle these into exe but use a different pkg file for changing py files?

For people who don't use computers well (non-technical people), pyinstaller lets them run python code -- I think such a feature would be a great addition to the community.

P.S. To be clear, all large libraries (numpy, pandas, etc.) are required, so I'm not looking for general shrinking tips (like UPX or excludes). The goal is specifically to split these static libs into a separate file for tiny updates. The 20MB email limit is a hard constraint.

Details

Here is my attempt:

    ENV_DIR = r'D:\Users\k2p\miniforge3\envs\my_app_env\Lib\site-packages' #HARDCODED 
    PYZ_EXTERNAL_TEMP_DIR = r"D:\my_temp_dir" #HARDCODED
    append_pkg = False

    pure_toc_mine_L = []
    pure_toc_libs_L = []
    for dst_name, src_path, typecode in a.pure:
        toc_tuple = (dst_name, src_path, typecode)
        if src_path.startswith(ENV_DIR):
            pure_toc_libs_L.append(toc_tuple)
        else:
            pure_toc_mine_L.append(toc_tuple)

    print('\n', f"{len(pure_toc_libs_L) = }", '\n')
    print('\n', f"{len(pure_toc_mine_L) = }", '\n')

    
    #### TRY PUT MINE IN EXE AND LIBS IN EXTERNAL (update with exe only)
    pyz_external_basename = f'pyz_external_{release_tk}.pyz' #must have pyz fext
    pyz_external_fullpath = os.path.join(PYZ_EXTERNAL_TEMP_DIR, pyz_external_basename)
    
    print(f"--- Building library PYZ at: {pyz_external_fullpath} ---") #PYZ __init__ makes the file on disk!
    pyz_mine = PYZ(pure_toc_mine_L, name=None) # autoname the one going into the EXE
    pyz_libs = PYZ(pure_toc_libs_L, name=pyz_external_fullpath) 
    pyz_for_exe, pyz_for_ext = pyz_mine, pyz_libs

    # Add the *path* to that new file to a.datas
    a.datas.append((pyz_external_basename, pyz_external_fullpath, 'PYZ'))

But i got this error:

Traceback (most recent call last):
  File "PyInstaller\hooks\rthooks\pyi_rth_pyqt5.py", line 67, in <module>
  File "PyInstaller\hooks\rthooks\pyi_rth_pyqt5.py", line 20, in _pyi_rthook
ModuleNotFoundError: No module named '_pyi_rth_utils'
[PYI-55224:ERROR] Failed to execute script 'pyi_rth_pyqt5' due to unhandled exception!

[rokm]

Is it easily possible to implement the use of multiple on-disk PKG files for the bundled app's start up sequence (boot strap / load)?

Not really.

The loading code expects the executable to have exactly one associated PKG archive, which is typically embedded in the executable. In onedir mode, the PKG contains bootstrap modules and scripts, run-time option entries, and the PYZ archive. In onefile mode, it also contains all data files and binaries that are extracted to the temporary directory at run-time.

The EXE class's keyword argument append_pkg=False hints to me that the machinery might already be there.

This option is for making the whole PKG archive external; i.e., instead of looking for PKG in the executable itself, the bootloader code looks for the .pkg file with matching basename that is located next to the executable. (The "append" refers to whether the PKG is appended to the executable file; it is not "append" in the sense of appending multiple entries).

But your question seems to be about PYZ archive, not the PKG one. The loading code also expects to find only one PYZ archive in the PKG archive - you can go out of your way to put multiple PYZ entries into PKG, but only the first will be used.

If you want to do incremental updates, you could go with onedir and forego the PYZ archive altogether; i.e., generate .spec with --debug noarchive option (or add a noarchive=True to Analysis() in the existing .spec). Then the .pyc modules will be collected as individual files and PYZ archive will be left empty.

But as far as I'm concerned, the proper solution to your updating problem would be to use some sort of differential updating scheme (i.e., patch the existing files with a differential patch to obtain the new full set). Perhaps see if you can find an installer that can do this for you.