Troubles using jnius in a multiprocess environment

[audetto]

Hi,

In the office we have tried to use jnius in a multiprocess environment.
This has been very problematic on linux where the whole thing hangs almost always.

In Windows it works.

It seems that if I set python's processes to spawn rather than fork, it works.

Java version does not seem to be relevant (1.8, 1.14, 1.17 all behave the same).
Python 3.6 and 3.8 are affected.

Is this supported at all?

[misl6]

Hi @audetto !

Do you have a reproducible example?

[audetto]

Will try, but so far I can only share a partial call stack

jnius has already been initialised in the main process, and the hang is in a forked process, not in the first call to autoclass, but probably the very first real java code executed.

[ramires9]

I would like to be able to use the jnius library in the renpy engine, for visual novels, and even install the files but I don't know where to put them, anywhere I try to put the non-regular renpy when importing

[lfse-slafleur]

Hi everyone! We were struggling with the same issue as we are using Celery in 'prefork' mode which spawns subprocesses. The 'fork vs. spawn' comment helped us out a bunch. The issue has to do with when pyjnius is loaded and with what process.

In essence pyjnius starts a Java VM when it is loaded. Now if you fork your process, the parent process and the child process gain the same file descriptors to communicate with this single Java VM. With 'spawn' both processes maintain their own file descriptors. Now I am unsure if a single Java VM from pyjnius with multiple 'spawn' subprocesses will work as then multiple processes are communicating with the same VM.

In our case, we fixed it by letting the subprocess load pyjnius after the fork/spawn and only then autoload the classes. This fixes it regardless if 'fork' or 'spawn' method is used for subprocesses. A snippet of what we used underneath. Hopefully this helps anyone else with dealing with this issue! So guideline: start up only 1 Java VM per process and pyjnius starts a Java VM when it is loaded.

The snippet:

class PyjniusLoader:
    INSTANCE = None
    loaded_classes: Dict[str, JavaClass]

    def __init__(self):
        path = os.path.dirname(__file__)
        import jnius_config  # noqa

        jnius_config.add_classpath(os.path.join(path, "some/path/here.jar"))

        self.loaded_classes = {}

    def load_class(self, classpath: str) -> JavaClass:
        from jnius import autoclass  # noqa
        if classpath not in self.loaded_classes:
            self.loaded_classes[classpath] = autoclass(classpath)

        return self.loaded_classes[classpath]

    @staticmethod
    def get_loader() -> 'PyjniusLoader':
        if PyjniusLoader.INSTANCE is None:
            PyjniusLoader.INSTANCE = PyjniusLoader()
        return PyjniusLoader.INSTANCE

And then of course only call PyjniusLoader.get_loader() from the subprocess. (or any process after all forking/spawning is done)