Unable to optimize or load parameters from TemplateExpressionSpec with guesses (v2.0.0a1)

[debraaudus]

Hello,

I was excited to see the guesses functionality in v2.0.0a1. When one has both a TemplateExpressionSpec and parameters, how can one either (a) specify the value of the parameters or (b) have them be optimized? An example with some basic debugging is below.

spec = pysr.TemplateExpressionSpec(
        expressions=["f"],
        variable_names=["x", "class"],
        parameters={"p1": 2},
        combine="f(x, p1[class])"
        )
modelsr = pysr.PySRRegressor(
        niterations=0,
        binary_operators=["+", "*", "-", "/"],
        guesses = [{"f" : "(#1 / #2)"}]
        )

The above will generate an equation of f = (#1 / #2); p1 = [0.0, 0.0]. In this case, this means an infinite loss.

guesses = [{"f" : "(#1 / #2)", "p1": [1.0, 2.0]}] yields an error.

Thank you so much!

[MilesCranmer]

Ah, interesting edge case! Yes so guesses for templates that have parameters will auto-init all parameters to zero. And it seems in your case this resulted in NaN guesses! So a quick workaround is make your guess equal to: #1 / (#2 + 1) and that should fix it, hopefully.

[MilesCranmer]

Actually it should automatically run constant optimization immediately after loading the guesses: https://github.com/astroautomata/SymbolicRegression.jl/blob/1728304ec9e8a60f5552dad62ead0b6026171b06/src/SearchUtils.jl#L832, no tricks needed. So for some reason the constant optimization is not having luck with your dataset. Maybe try checking if you can manually optimize your guess expression (with Optim.jl directly, for example), or maybe something about the loss surface makes this exceedingly difficult? e.g., maybe its worth trying a Simplex method (NelderMead) instead of the default BFGS.

[MilesCranmer]

Oh wait... I wonder...

@MilesCranmerBot can you check if the SR.jl version used by PySR 2.0.0a1 is missing the logic that will immediately optimize template expression guesses after parsing? Maybe it's only PySR 2.0.0a2?

[debraaudus]

I'm not quite sure what's going on, but here's what I've learned. Yes, you are right that there is code to do constant optimization immediately after the equation is loaded; I checked all the versions of the julia code. However, there's something a bit odd about the constant optimization. Here are three cases:

(1) guesses = [{"f" : "(#1 / #2)"}] does not do parameter optimization. This can be verified since the output has all parameters set to 0.

(2) guesses = [{"f" : "(#1 / (#2 + 0.0001))"}] does both constant and parameter optimization. Success!

(3) guesses = [{"f" : "(#1 / (#2 + 0))"}] does not do parameter optimization. This can be verified since the output has all parameters set to 0.

So, technically, a workaround is to add a very small constant. This is not ideal as the complexity has now increased by 2, but it is still useful as the potential for finding the right thing is much higher now.

[MilesCranmer]

This makes sense to me; it's because the (#1 / #2) and (#1 / (#2 + 0)) are both NaN at initialization (since init param values are 0), so the optimization can't even proceed. So offsetting by the 0.0001 constant is actually the right move here. And it should actually remove that constant once it has found the parameters, so this might not even matter in the end

[debraaudus]

Thank you so much! I just checked guesses = [{"f" : "#2"}] and that works, so I think I just have to be careful with equations where initial parameters of 0 give a loss of NaN.