[BUG]: Using cluster manager as "slurm" is not working

[anirudh-nemmani]

What happened?

I started running PySR in a cluster with the following configuration.

PySRRegressor(
        binary_operators=["+", "-", "*"],
        unary_operators=["square", "cube", "fourth(x) = x^4"],
        extra_sympy_mappings={"fourth": lambda x: x**4},
        expression_spec=template,
        niterations=100,
        batching=True,
        batch_size=4096,
        populations=150,
        population_size=128,
        maxdepth=7,
        maxsize=32,
        parsimony=1e-4,
        elementwise_loss="L2DistLoss()",
        procs=30,
        cluster_manager="slurm",
        parallelism="multiprocessing",
        turbo=True,
        ncycles_per_iteration=100,
    )

And when I submit it in Slurm with the following Slurm settings

#!/bin/bash -l
#SBATCH -J SymQuark-Mass
#SBATCH -N 1
#SBATCH --ntasks-per-node=30
#SBATCH --nodelist=chuck-[29-34]
#SBATCH --mem=16G
#SBATCH --time=1-00:00:00
#SBATCH -p para
#SBATCH --output=logs/log.out
#SBATCH --error=logs/log.err

cd $SLURM_SUBMIT_DIR

module add mpi/mvapich2-x86_64

source <path-to-conda>
conda activate pysr

python mass-symbolic.py

When I run with this configuration, I am noticing that in each worker, when Julia is initialised and the code is running, it gives an error during compilation.

juliacall.JuliaError: On worker 2:
UndefVarError: `ComposableExpressionModule` not defined in `SymbolicRegression`

I attached the entire logs also

Version

1.5.9

Operating System

Linux

Package Manager

Conda

Interface

Script (i.e., python3 script.py)

Relevant log output

cpu-bind=MASK - chuck-29, task  0  0 [6082]: mask 0x3fffffff set
Compiling Julia backend...
Traceback (most recent call last):
  File "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/mass-symbolic.py", line 623, in <module>
    main()
    ~~~~^^
  File "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/mass-symbolic.py", line 604, in main
    model.fit(train_x, train_y)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/anemmani/softwares/anaconda3/envs/pysr/lib/python3.14/site-packages/pysr/sr.py", line 2318, in fit
    self._run(X, y, runtime_params, weights=weights, seed=seed, category=category)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anemmani/softwares/anaconda3/envs/pysr/lib/python3.14/site-packages/pysr/sr.py", line 2109, in _run
    out = SymbolicRegression.equation_search(
        jl_X,
    ...<27 lines>...
        logger=logger,
    )
  File "/home/anemmani/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl", line 262, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juliacall.JuliaError: On worker 2:
UndefVarError: `ComposableExpressionModule` not defined in `SymbolicRegression`
Stacktrace:
  [1] deserialize_module
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1034
  [2] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:933
  [3] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
  [4] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1543
  [5] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
  [6] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
  [7] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1555
  [8] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
  [9] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
 [10] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1568
 [11] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
 [12] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
 [13] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1559
 [14] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
 [15] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
 [16] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1568
 [17] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
 [18] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
 [19] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:911
 [20] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851 [inlined]
 [21] deserialize_msg
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/messages.jl:87
 [22] message_handler_loop
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:176
 [23] process_tcp_streams
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:133
 [24] #process_messages##0
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:121
Stacktrace:
  [1] #remotecall_fetch#109
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:465
  [2] remotecall_fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:454
  [3] remotecall_fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:492 [inlined]
  [4] call_on_owner
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:565 [inlined]
  [5] fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:619
  [6] test_function_on_workers(example_inputs::Tuple{@NamedTuple{f::ComposableExpression{Float64, Node{Float64}, @NamedTuple{operators::OperatorEnum{Tuple{typeof(+), typeof(-), typeof(*)}, Tuple{typeof(square), typeof(cube), typeof(fourth)}}, variable_names::Nothing, eval_options::EvalOptions{true, false, true, Nothing}}}}, @NamedTuple{p1::SymbolicRegression.TemplateExpressionModule.ParamVector{Float64}, p2::SymbolicRegression.TemplateExpressionModule.ParamVector{Float64}}, Vector{ValidVector{Vector{Float64}}}}, op::typeof(__sr_template_2337148491627256653), procs::Vector{Int64})
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/Configure.jl:237
  [7] move_functions_to_workers(procs::Vector{Int64}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum{Tuple{typeof(+), typeof(-), typeof(*)}, Tuple{typeof(square), typeof(cube), typeof(fourth)}}, Node, TemplateExpression, @NamedTuple{structure::TemplateStructure{(:f,), (:p1, :p2), typeof(__sr_template_2337148491627256653), @NamedTuple{f::Int64}, @NamedTuple{p1::Int64, p2::Int64}}}, MutationWeights, true, false, nothing, Nothing, 8}, dataset::SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}, verbosity::Int64)
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/Configure.jl:211
  [8] #configure_workers#15
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/Configure.jl:389 [inlined]
  [9] _create_workers(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multiprocessing, 1, true, Nothing}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum{Tuple{typeof(+), typeof(-), typeof(*)}, Tuple{typeof(square), typeof(cube), typeof(fourth)}}, Node, TemplateExpression, @NamedTuple{structure::TemplateStructure{(:f,), (:p1, :p2), typeof(__sr_template_2337148491627256653), @NamedTuple{f::Int64}, @NamedTuple{p1::Int64, p2::Int64}}}, MutationWeights, true, false, nothing, Nothing, 8})
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:633
 [10] _equation_search(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multiprocessing, 1, true, Nothing}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum{Tuple{typeof(+), typeof(-), typeof(*)}, Tuple{typeof(square), typeof(cube), typeof(fourth)}}, Node, TemplateExpression, @NamedTuple{structure::TemplateStructure{(:f,), (:p1, :p2), typeof(__sr_template_2337148491627256653), @NamedTuple{f::Int64}, @NamedTuple{p1::Int64, p2::Int64}}}, MutationWeights, true, false, nothing, Nothing, 8}, saved_state::Nothing)
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:568
 [11] equation_search(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}; options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum{Tuple{typeof(+), typeof(-), typeof(*)}, Tuple{typeof(square), typeof(cube), typeof(fourth)}}, Node, TemplateExpression, @NamedTuple{structure::TemplateStructure{(:f,), (:p1, :p2), typeof(__sr_template_2337148491627256653), @NamedTuple{f::Int64}, @NamedTuple{p1::Int64, p2::Int64}}}, MutationWeights, true, false, nothing, Nothing, 8}, saved_state::Nothing, runtime_options::Nothing, runtime_options_kws::@Kwargs{niterations::Int64, parallelism::String, numprocs::Int64, procs::Nothing, addprocs_function::typeof(addprocs_slurm), heap_size_hint_in_bytes::Nothing, worker_imports::Nothing, runtests::Bool, return_state::Bool, run_id::String, verbosity::Int64, logger::Nothing, progress::Bool, v_dim_out::Val{1}})
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:561
 [12] equation_search
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:542 [inlined]
 [13] #equation_search#16
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:511 [inlined]
 [14] equation_search
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:456 [inlined]
 [15] #equation_search#17
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:535 [inlined]
 [16] pyjlany_call(self::typeof(equation_search), args_::Py, kwargs_::Py)
    @ PythonCall.JlWrap ~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:44
 [17] _pyjl_callmethod
    @ ~/.julia/packages/PythonCall/avYrV/src/JlWrap/base.jl:73
 [18] _pyjl_callmethod
    @ ~/.julia/packages/PythonCall/avYrV/src/JlWrap/C.jl:63
[ Info: Automatically setting `--heap-size-hint=4169M` on each Julia process. You can configure this with the `heap_size_hint_in_bytes` parameter.
┌ Warning: The Slurm functionality in the `ClusterManagers.jl` package is deprecated (including `ClusterManagers.addprocs_slurm` and `ClusterManagers.SlurmManager`). It will be removed from ClusterManagers.jl in a future release. We recommend migrating to the [https://github.com/JuliaParallel/SlurmClusterManager.jl](https://github.com/JuliaParallel/SlurmClusterManager.jl) package instead.
│   caller = ip:0x0
└ @ Core :-1
[ Info: Starting SLURM job julia-6153: `srun -J julia-6153 -n 30 -D /work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params -o /work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/./julia-6153-17684729876-%4t.out /home/anemmani/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/bin/julia --heap-size=4169M --worker=AG58K97zmQeSwBlt`
[ Info: Worker 0 (after 0 s): No output file "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/./julia-6153-17684729876-0000.out" yet
[ Info: Worker 0 (after 1 s): Output file found, but no connection details yet
[ Info: Worker 0 ready after 3 s on host 192.168.6.29, port 9816
[ Info: Worker 1 ready after 3 s on host 192.168.6.29, port 9817
[ Info: Worker 2 ready after 3 s on host 192.168.6.29, port 9818
[ Info: Worker 3 ready after 3 s on host 192.168.6.29, port 9819
[ Info: Worker 4 ready after 3 s on host 192.168.6.29, port 9820
[ Info: Worker 5 ready after 3 s on host 192.168.6.29, port 9821
[ Info: Worker 6 ready after 3 s on host 192.168.6.29, port 9822
[ Info: Worker 7 ready after 3 s on host 192.168.6.29, port 9823
[ Info: Worker 8 ready after 3 s on host 192.168.6.29, port 9824
[ Info: Worker 9 ready after 3 s on host 192.168.6.29, port 9825
[ Info: Worker 10 ready after 3 s on host 192.168.6.29, port 9826
[ Info: Worker 11 ready after 3 s on host 192.168.6.29, port 9827
[ Info: Worker 12 ready after 3 s on host 192.168.6.29, port 9828
[ Info: Worker 13 ready after 3 s on host 192.168.6.29, port 9829
[ Info: Worker 14 ready after 3 s on host 192.168.6.29, port 9830
[ Info: Worker 15 ready after 3 s on host 192.168.6.29, port 9831
[ Info: Worker 16 ready after 3 s on host 192.168.6.29, port 9832
[ Info: Worker 17 ready after 3 s on host 192.168.6.29, port 9833
[ Info: Worker 18 ready after 3 s on host 192.168.6.29, port 9834
[ Info: Worker 19 ready after 3 s on host 192.168.6.29, port 9835
[ Info: Worker 20 ready after 3 s on host 192.168.6.29, port 9836
[ Info: Worker 21 ready after 3 s on host 192.168.6.29, port 9837
[ Info: Worker 22 ready after 3 s on host 192.168.6.29, port 9838
[ Info: Worker 23 ready after 3 s on host 192.168.6.29, port 9839
[ Info: Worker 24 ready after 3 s on host 192.168.6.29, port 9840
[ Info: Worker 25 ready after 3 s on host 192.168.6.29, port 9841
[ Info: Worker 26 ready after 3 s on host 192.168.6.29, port 9842
[ Info: Worker 27 ready after 3 s on host 192.168.6.29, port 9843
[ Info: Worker 28 ready after 3 s on host 192.168.6.29, port 9844
[ Info: Worker 29 ready after 3 s on host 192.168.6.29, port 9845
[ Info: Importing SymbolicRegression on workers as well as extensions ClusterManagers, LoopVectorization.
[ Info: Finished!
[ Info: Copying definition of fourth to workers...
[ Info: Finished!
[ Info: Copying definition of __sr_template_2337148491627256653 to workers...
[ Info: Finished!

Extra Info

No response

[MilesCranmer]

If you are using only one node, you don't need to use the cluster manager, it should work with normal settings.

The error is weird but might be from conflicting Julia installations somewhere in your system

[anirudh-nemmani]

It seems there's only one Julia installation on my system, and it's version 1.12.3. From the error log, it appears to be used correctly. The same script, when ran in the terminal instead of via job submission, works perfectly.

[anirudh-nemmani]

Thank you for the response!

This is a test case where I am running only on one node. But I intend to use multiple nodes.

Regarding the error, it is indeed weird; maybe you might be right regarding the multiple Julia installations. I will try to fix that and see if it will work.

[MilesCranmer]

Oh wait. I think I might know what it is and it could be a real bug. I just noticed you are using a template expression. Can you share your template expression definition?

I think it could be that the template expression function is not getting defined on the worker nodes. I've never actually tried using template expressions in a distributed setup before.

[anirudh-nemmani]

The template I am using is this

template = TemplateExpressionSpec(
        expressions=["f"],
        variable_names=["x1", "x2"],
        parameters={"p1": 1},
        combine=("p1[1] * (2*x2/(x2 + 1))^(2/3) * f(x1, x2)"),
    )

Maybe you are right. I tried running the script without a template expression, but I got a different error now.

cpu-bind=MASK - chuck-29, task  0  0 [6086]: mask 0x3fffffff set
Compiling Julia backend...
Traceback (most recent call last):
  File "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/mass-symbolic.py", line 623, in <module>
    main()
    ~~~~^^
  File "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/mass-symbolic.py", line 604, in main
    model.fit(train_x, train_y)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/anemmani/softwares/anaconda3/envs/pysr/lib/python3.14/site-packages/pysr/sr.py", line 2318, in fit
    self._run(X, y, runtime_params, weights=weights, seed=seed, category=category)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/anemmani/softwares/anaconda3/envs/pysr/lib/python3.14/site-packages/pysr/sr.py", line 2109, in _run
    out = SymbolicRegression.equation_search(
        jl_X,
    ...<27 lines>...
        logger=logger,
    )
  File "/home/anemmani/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl", line 262, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
juliacall.JuliaError: On worker 2:
TypeError: in Type{...} expression, expected UnionAll, got Type{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, VC} where VC<:Union{Int64, AbstractVector{Int64}}}
Stacktrace:
  [1] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1568
  [2] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
  [3] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
  [4] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1568
  [5] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
  [6] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
  [7] deserialize_datatype
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:1568
  [8] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:904
  [9] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851
 [10] handle_deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:911
 [11] deserialize
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:851 [inlined]
 [12] deserialize_msg
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/messages.jl:87
 [13] message_handler_loop
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:176
 [14] process_tcp_streams
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:133
 [15] #process_messages##0
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/process_messages.jl:121
Stacktrace:
  [1] #remotecall_fetch#109
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:465
  [2] remotecall_fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:454
  [3] remotecall_fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:492 [inlined]
  [4] call_on_owner
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:565 [inlined]
  [5] fetch
    @ ~/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:619
  [6] test_module_on_workers(procs::Vector{Int64}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum, Node, Expression, @NamedTuple{}, MutationWeights, true, false, nothing, Nothing, 8}, verbosity::Int64)
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/Configure.jl:321
  [7] #configure_workers#15
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/Configure.jl:392 [inlined]
  [8] _create_workers(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multiprocessing, 1, true, Nothing}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum, Node, Expression, @NamedTuple{}, MutationWeights, true, false, nothing, Nothing, 8})
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:633
  [9] _equation_search(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}, ropt::SymbolicRegression.SearchUtilsModule.RuntimeOptions{:multiprocessing, 1, true, Nothing}, options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum, Node, Expression, @NamedTuple{}, MutationWeights, true, false, nothing, Nothing, 8}, saved_state::Nothing)
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:568
 [10] equation_search(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.BasicDataset{Float64, Float64, Matrix{Float64}, Vector{Float64}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}}; options::Options{SymbolicRegression.CoreModule.OptionsStructModule.ComplexityMapping{Int64, Int64}, OperatorEnum, Node, Expression, @NamedTuple{}, MutationWeights, true, false, nothing, Nothing, 8}, saved_state::Nothing, runtime_options::Nothing, runtime_options_kws::@Kwargs{niterations::Int64, parallelism::String, numprocs::Int64, procs::Nothing, addprocs_function::typeof(addprocs_slurm), heap_size_hint_in_bytes::Nothing, worker_imports::Nothing, runtests::Bool, return_state::Bool, run_id::String, verbosity::Int64, logger::Nothing, progress::Bool, v_dim_out::Val{1}})
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:561
 [11] equation_search
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:542 [inlined]
 [12] #equation_search#16
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:511 [inlined]
 [13] equation_search
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:456 [inlined]
 [14] #equation_search#17
    @ ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:535 [inlined]
 [15] pyjlany_call(self::typeof(equation_search), args_::Py, kwargs_::Py)
    @ PythonCall.JlWrap ~/.julia/packages/PythonCall/avYrV/src/JlWrap/any.jl:44
 [16] _pyjl_callmethod
    @ ~/.julia/packages/PythonCall/avYrV/src/JlWrap/base.jl:73
 [17] _pyjl_callmethod
    @ ~/.julia/packages/PythonCall/avYrV/src/JlWrap/C.jl:63
[ Info: Automatically setting `--heap-size-hint=4171M` on each Julia process. You can configure this with the `heap_size_hint_in_bytes` parameter.
┌ Warning: The Slurm functionality in the `ClusterManagers.jl` package is deprecated (including `ClusterManagers.addprocs_slurm` and `ClusterManagers.SlurmManager`). It will be removed from ClusterManagers.jl in a future release. We recommend migrating to the [https://github.com/JuliaParallel/SlurmClusterManager.jl](https://github.com/JuliaParallel/SlurmClusterManager.jl) package instead.
│   caller = ip:0x0
└ @ Core :-1
[ Info: Starting SLURM job julia-6156: `srun -J julia-6156 -n 30 -D /work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params -o /work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/./julia-6156-17685659687-%4t.out /home/anemmani/.julia/juliaup/julia-1.12.3+0.x64.linux.gnu/bin/julia --heap-size=4171M --worker=f7zhw4F0YRW6YYQl`
[ Info: Worker 0 (after 0 s): No output file "/work/chuck/anemmani/git-repos/symbolic-quark/scripts/internal-params/./julia-6156-17685659687-0000.out" yet
[ Info: Worker 0 (after 1 s): Output file found, but no connection details yet
[ Info: Worker 0 ready after 3 s on host 192.168.6.29, port 9826
[ Info: Worker 1 ready after 3 s on host 192.168.6.29, port 9827
[ Info: Worker 2 ready after 3 s on host 192.168.6.29, port 9828
[ Info: Worker 3 ready after 3 s on host 192.168.6.29, port 9829
[ Info: Worker 4 ready after 3 s on host 192.168.6.29, port 9830
[ Info: Worker 5 ready after 3 s on host 192.168.6.29, port 9831
[ Info: Worker 6 ready after 3 s on host 192.168.6.29, port 9832
[ Info: Worker 7 ready after 3 s on host 192.168.6.29, port 9833
[ Info: Worker 8 ready after 3 s on host 192.168.6.29, port 9834
[ Info: Worker 9 ready after 3 s on host 192.168.6.29, port 9835
[ Info: Worker 10 ready after 3 s on host 192.168.6.29, port 9836
[ Info: Worker 11 ready after 3 s on host 192.168.6.29, port 9837
[ Info: Worker 12 ready after 3 s on host 192.168.6.29, port 9838
[ Info: Worker 13 ready after 3 s on host 192.168.6.29, port 9839
[ Info: Worker 14 ready after 3 s on host 192.168.6.29, port 9840
[ Info: Worker 15 ready after 3 s on host 192.168.6.29, port 9841
[ Info: Worker 16 ready after 3 s on host 192.168.6.29, port 9842
[ Info: Worker 17 ready after 3 s on host 192.168.6.29, port 9843
[ Info: Worker 18 ready after 3 s on host 192.168.6.29, port 9844
[ Info: Worker 19 ready after 3 s on host 192.168.6.29, port 9845
[ Info: Worker 20 ready after 3 s on host 192.168.6.29, port 9846
[ Info: Worker 21 ready after 3 s on host 192.168.6.29, port 9847
[ Info: Worker 22 ready after 3 s on host 192.168.6.29, port 9848
[ Info: Worker 23 ready after 3 s on host 192.168.6.29, port 9849
[ Info: Worker 24 ready after 3 s on host 192.168.6.29, port 9850
[ Info: Worker 25 ready after 3 s on host 192.168.6.29, port 9851
[ Info: Worker 26 ready after 3 s on host 192.168.6.29, port 9852
[ Info: Worker 27 ready after 3 s on host 192.168.6.29, port 9853
[ Info: Worker 28 ready after 3 s on host 192.168.6.29, port 9854
[ Info: Worker 29 ready after 3 s on host 192.168.6.29, port 9855
[ Info: Importing SymbolicRegression on workers as well as extensions ClusterManagers, LoopVectorization.
[ Info: Finished!
[ Info: Copying definition of fourth to workers...
[ Info: Finished!
[ Info: Testing module on workers...

[MilesCranmer]

Argh. Sorry about this. Any chance you could try Julia 1.11? The new release added some changes to compiled functions that might be making it more challenging to distribute runtime generated types.

Really what we need in PySR is a proper way to unit test slurm code (via docker composer). I started working on this but haven't had the chance to finish it sadly.

[anirudh-nemmani]

Regarding the Julia version, I noticed this in some other discussions also. I am new to Julia, so correct me if I am wrong. I added a new Julia version in my system and then made it default to 1.11. But when I run PySR, it takes the newer version of Julia. Does PySR has a specific Julia version manager to pick the Julia version?

Yes having a proper docker setup for the slurm code is really helpful. I am also waiting for your fix for Issue #941 xD.

[MilesCranmer]

@anirudh-nemmani you can ask for a specific Julia version like this:

import juliapkg

juliapkg.require_julia("~1.11")

# Make sure to import pysr only afterwards
import pysr

[anirudh-nemmani]

It seems that in Julia 1.11, both cases (with and without a template expression) work. So this has something to do with Julia 1.12, maybe?

Something really weird.

Everything works perfectly except for this warning in the end. I am assuming this is Julia closing all the workers that were initialised.

UNHANDLED TASK ERROR: Distributed.ProcessExitedException(20)
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:958
  [2] wait()
    @ Base ./task.jl:1022
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_buffered(c::Channel{Any})
    @ Base ./channels.jl:493
  [6] take!(c::Channel{Any})
    @ Base ./channels.jl:487
  [7] take!(::Distributed.RemoteValue)
    @ Distributed ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:726
  [8] remotecall_fetch(f::Function, w::Distributed.Worker, args::Distributed.RRID; kwargs::@Kwargs{})
    @ Distributed ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:461
  [9] remotecall_fetch(f::Function, w::Distributed.Worker, args::Distributed.RRID)
    @ Distributed ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:454
 [10] remotecall_fetch
    @ ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:492 [inlined]
 [11] call_on_owner
    @ ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:565 [inlined]
 [12] fetch
    @ ~/.julia/juliaup/julia-1.11.8+0.x64.linux.gnu/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:619
 [13] (::SymbolicRegression.var"#63#68"{SymbolicRegression.SearchUtilsModule.SearchState{Float64, Float64, Expression{Float64, Node{Float64}, @NamedTuple{operators::Nothing, variable_names::Nothing}}, Distributed.Future, Distributed.RemoteChannel}, Int64, Int64})()
    @ SymbolicRegression ~/.julia/packages/SymbolicRegression/L5TJa/src/SymbolicRegression.jl:976
    ```

[MilesCranmer]

@anirudh-nemmani can you please try installing the master branch version of pysr? I fixed some slurm issues (and now actually have testing!!) in #1111. I'm curious if it fixes the issues you saw.

[anirudh-nemmani]

Hey! Sorry for the late reply . I was busy with some other things. Sure I can try to test this version of the code, this week. I will let you know by tomorrow.