Add Lapack dependencies to meson.build for XSF

[brorson]

I am working on an implementation of the Mathieu functions for xsf. The current impls have a number of bugs which I have fixed. @steppi and @lucascolley are aware of this work.

I have added my code to my GitHub repo https://github.com/brorson/xsf_mathieu. That's my fork of the xsf repo.

However, to do a build I also need to update the build dependencies for xsf. The meson.build file for xsf lives in the main Scipy trunk, not in the xsf subproject. Therefore, I am requesting this pull which will update the meson.build file so the xsf stuff will build.

Important note: Once the meson.build and the xsf stuff are put together and a build is performed, the CI tests will fail. See my test failures below. The tests fail since the old tests just checked some golden values generated from the buggy, old impl. The new impl returns correct values which are different from the old one so failing tests are expected.

I have students working on creating a Python test suite for this Mathieu fcn impl and will be happy to add that to Scipy once it is ready. If desired, you could create a branch where we could work to finalize the Mathieu implementation.

My pull requests should close these bugs (and others too):
#4479
#14526
https://stackoverflow.com/questions/28205127/mathieu-characteristics-cross-when-plotted
#14577

Failed tests:
================================================== short test summary info ==================================================
FAILED scipy/special/tests/test_basic.py::TestCephes::test_mathieu_cem - AssertionError:
FAILED scipy/special/tests/test_basic.py::TestCephes::test_mathieu_sem - AssertionError:
FAILED scipy/special/tests/test_basic.py::TestCephes::test_mathieu_modcem2 - AssertionError:
FAILED scipy/special/tests/test_basic.py::TestCephes::test_mathieu_modsem2 - AssertionError:
FAILED scipy/special/tests/test_basic.py::TestCephes::test_mathieu_ticket_1847 - AssertionError:
FAILED scipy/special/tests/test_data.py::test_gsl[] - AssertionError:
FAILED scipy/special/tests/test_data.py::test_gsl[] - AssertionError:
FAILED scipy/special/tests/test_data.py::test_gsl[] - AssertionError:
FAILED scipy/special/tests/test_data.py::test_gsl[] - AssertionError:
FAILED scipy/special/tests/test_data.py::test_gsl[] - AssertionError:
FAILED scipy/special/tests/test_specfun.py::test_cva2_cv0_branches - AssertionError:
=================== 11 failed, 7673 passed, 421 skipped, 184 deselected, 54 xfailed, 1 xpassed in 44.50s ====================

[rgommers]

Thanks for working on this @brorson! This is for scipy/xsf#60, right? I see the dsyev usage there.

This change looks correct, but we shouldn't merge it separate from updating the xsf git submodule that needs this. I think the correct thing to do is:

• change this PR to Draft status
• add an xsf git submodule update in this PR, pointing at the most recent commit in your branch with the mathieu improvements
• once your xsf PR is merged, update it to point at the merge commit in the main branch of the xsf repo
• merge this PR once CI here is happy.

Does that work for you?

[brorson]

Sorry for the late reply. Yes, your proposed plan of action is fine with me. The issue I encountered is that my code changes are in the xsf repo while the necessary changes to the build system are in the main scipy repo. It looks like the scipy stuff grabs the latest xsf stuff from the xsf repo when doing a checkout, but I don't understand how they are linked. That's my inexperience with the GitHub workflow.

My goal is to include the build system changes along with the code changes in xsf/include/xsf. Whatever mechanism you have to achieve that is fine with me.

[rgommers]

My goal is to include the build system changes along with the code changes in xsf/include/xsf. Whatever mechanism you have to achieve that is fine with me.

It's always a little annoying to work on code that's split across repos. The way I usually do this is to work on the git submodule (xsf in this case) from inside the main project's repo rather than in a separate fork. So in this case:

1. Create a branch named mathieu in my local clone of the scipy repo (not in main like you did, that's always a recipe for issues later on)
2. Make the changing to the meson.build files you need
3. cd subprojects/xsf
4. Make a new branch also called mathieu inside that xsf submodule
5. Implement and commit the changes to xsf you want. You can push those to brorson/xsf and make a PR to xsf and add new commits to it.
6. If you need more changes to SciPy or want to include your xsf changes in this PR: cd ../.. (back to root of the repo), then git commit will now include both your scipy and xsf changes. The diff on this PR will just show a different commit hash for the xsf changes.

[brorson]

Thank you for the suggestion. I will try that -- I appreciate the advice since I have not dealt with this type of GitHub workflow before. I am busy with other stuff right now so I won't be able to get to it until next week, but please remain patient -- my goal is to see this through to completion.

[rgommers]

Sure, no hurry at all - please do ask if you need another pointer or run into an issue. We only recently split out xsf, and we should probably document this in the xsf README.

[rgommers]

I'll xref @steppi's plan for dealing with LAPACK in special functions here: #24871 (comment)

This PR probably isn't viable anymore, but I'm not 100% sure so I'll leave it open for the two of you to decide.

[brorson]

Yes, you can close this (it already has been closed).

Regarding Mathieu, I have a pull request in XSF which takes care of the LAPACK problem and provides the corrected Mathieu implementation. I found a way to point to LAPACK into the XSF build so my Mathieu impl could find and link it. The pull request is here:

scipy/xsf#99

I have been intending to ping the SciPy team about this pull request since I believe I completed the implementation and provided lots of documentation regarding how it works. The impl doesn't pass the CI tests since the CI tests (particularly the Parquet ones) were made to just test the old impl against itself. I stand ready to help get past the CI gate but likely need somebody from the SciPy or XSF team to work with me on it (i.e. somebody with access permissions). But at this point I am done with the Mathieu work and am just waiting for SciPy to pick it up (if desired).

Regarding Steppi's recommendation for ufuncs to avoid LAPACK calls and go through a gufunc, I am not familiar with how ufuncs and gufuncs work -- they are deep inside the SciPy's clockwork and all I did was replace the old scalar entry point to Mathieu with a different scalar entry point. I am ready to help out if desired but would need some pointers about how the ufunc stuff works or how one should use it with the underlying scalar C++ codes.

Thanks,
Stuart Brorson
Northeastern University

[steppi]

Regarding Steppi's recommendation for ufuncs to avoid LAPACK calls and go through a gufunc, I am not familiar with how ufuncs and gufuncs work -- they are deep inside the SciPy's clockwork and all I did was replace the old scalar entry point to Mathieu with a different scalar entry point. I am ready to help out if desired but would need some pointers about how the ufunc stuff works or how one should use it with the underlying scalar C++ codes.

I expect the refactoring plan I suggested should be fairly straightforward for me to do now that you've completed all of the heavy lifting writing a much cleaner and better C++ implementation, since I have experience with ufunc and gufunc internals. I'll help with that part.