Additional modularization of stdlib

[jvdp1]

Update:

• All files in src (except stdlib_version.fypp) were moved to the follwing subfolders (a library is created within each subfolder): ansi, bitsets, core, hashmaps, io, lapack_extended linalg_core math selection specialfunctions stats, stringlist system array blas constants hash intrinsics lapack linalg logger quadrature sorting specialmatrices strings, sparse and linalg_iterative
• This structure closely follows the structure in test and examples.
• The folders core and linalg_core contain modules that are required by various parts of stdlib and were created to avoid interdependencies.
• The file stdlib_version.fypp was maintained in src to avoid to modify cmake/stdlib.cmake in this PR. It could be moved later to the subfolder core.
• Optional compilaton (like for stdlib_ansi and stdlib_hashmaps` will be added in a following PR.

Depends on #1088

[jalvesz]

@jvdp1 I wonder if it would be advisable given all different options being added to include in the readme.md a table listing all possible macros/flags. I asked copilot to generate a first version that would look like this:

### Preprocessing macros / flags

`stdlib` uses two preprocessing steps:

- **fypp** for meta-programming (templating and feature selection)
- **C preprocessing** (compiler `-cpp`/`-fpp`) for conditional compilation

The table below lists all preprocessing macros/flags currently used by `stdlib`.

| Macro/flag Name | preprocessing | comments |
| --- | --- | --- |
| `MAXRANK` | fypp | Maximum array rank generated by templates. Set via CMake `-DCMAKE_MAXIMUM_RANK=<n>` (passed to fypp as `-DMAXRANK=<n>`), via `fpm.toml` `preprocess.fypp.macros=["MAXRANK=<n>"]`, or via `python config/fypp_deployment.py --maxrank <n>`. |
| `VERSION90` | fypp | Enables a more Fortran-90-friendly preprocessing mode (affects defaults such as the maximum rank generated). Can be passed directly to fypp as `-DVERSION90` (CMake sets this automatically in some configurations). |
| `WITH_CBOOL` | fypp | Enables `c_bool` logical support if available. CMake auto-detects this and passes it to fypp; can be overridden at configure time with `-DWITH_CBOOL=ON/OFF`. |
| `WITH_QP` | fypp | Enables quadruple precision code generation (`real(qp)`, `complex(qp)`). CMake: `-DWITH_QP=ON/OFF`; fypp deployment script: `--with_qp`. |
| `WITH_XDP` | fypp | Enables extended double precision code generation (`real(xdp)`, `complex(xdp)`). CMake: `-DWITH_XDP=ON/OFF`; fypp deployment script: `--with_xdp`. |
| `WITH_ILP64` | fypp | Enables generation of 64-bit integer (ILP64) linear-algebra interfaces (built in addition to the default 32-bit interface). CMake: `-DWITH_ILP64=ON/OFF`; fypp deployment script: `--with_ilp64`. |
| `PROJECT_VERSION_MAJOR` | fypp | Part of the version string passed into fypp templates. Set automatically by CMake/from `VERSION`, but can be overridden by passing `-DPROJECT_VERSION_MAJOR=<n>` (used by `config/fypp_deployment.py`). |
| `PROJECT_VERSION_MINOR` | fypp | See `PROJECT_VERSION_MAJOR`. |
| `PROJECT_VERSION_PATCH` | fypp | See `PROJECT_VERSION_MAJOR`. |
| `STDLIB_NO_BITSET` | C | Disables compilation of the bitsets component. CMake: `-DSTDLIB_NO_BITSET=ON`; or define for the compiler preprocessor (e.g., `-DSTDLIB_NO_BITSET`). This makes `STDLIB_BITSET` evaluate to 0 via `include/macros.inc`. |
| `STDLIB_BITSET` | C | Internal numeric macro (0/1) derived from `STDLIB_NO_BITSET` in `include/macros.inc`. Used in code as `#if STDLIB_BITSET == 1`. Prefer setting `STDLIB_NO_BITSET` rather than defining `STDLIB_BITSET` directly. |
| `STDLIB_NO_STATS` | C | Disables compilation of the statistics component. CMake: `-DSTDLIB_NO_STATS=ON`; or define for the compiler preprocessor (e.g., `-DSTDLIB_NO_STATS`). This makes `STDLIB_STATS` evaluate to 0 via `include/macros.inc`. |
| `STDLIB_STATS` | C | Internal numeric macro (0/1) derived from `STDLIB_NO_STATS` in `include/macros.inc`. Used in code as `#if STDLIB_STATS == 1`. Prefer setting `STDLIB_NO_STATS` rather than defining `STDLIB_STATS` directly. |
| `STDLIB_EXTERNAL_BLAS` | C | Use an external BLAS (32-bit integer interface). Usually set by CMake when external BLAS/LAPACK are found, or manually via `add_compile_definitions(STDLIB_EXTERNAL_BLAS)`. In fpm: `preprocess.cpp.macros=["STDLIB_EXTERNAL_BLAS", ...]`. |
| `STDLIB_EXTERNAL_LAPACK` | C | Use an external LAPACK (32-bit integer interface). Usually paired with `STDLIB_EXTERNAL_BLAS`. |
| `STDLIB_EXTERNAL_BLAS_I64` | C | Use an external BLAS with ILP64 (64-bit integer) interfaces. Usually paired with `STDLIB_EXTERNAL_LAPACK_I64`. |
| `STDLIB_EXTERNAL_LAPACK_I64` | C | Use an external LAPACK with ILP64 (64-bit integer) interfaces. |

[jalvesz]

sorry @jvdp1 I was writting a comment and pushed on "Close" by accident, just reopened.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.55%. Comparing base (463821a) to head (3071410).
⚠️ Report is 56 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1081      +/-   ##
==========================================
+ Coverage   68.51%   68.55%   +0.03%     
==========================================
  Files         396      396              
  Lines       12746    12746              
  Branches     1376     1376              
==========================================
+ Hits         8733     8738       +5     
+ Misses       4013     4008       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jvdp1]

@jvdp1 I wonder if it would be advisable given all different options being added to include in the readme.md a table listing all possible macros/flags. I asked copilot to generate a first version that would look like this:

Thank you @jalvesz .I agree that such a table is needed. I will open a new PR based on your first version.

[jalvesz]

@jvdp1 after the issues encountered with #1086 it makes me think that we need to find a way of enforcing a strict hierarchy of dependencies to limit build time complexities.

The first level would be "a module at the root of src can depend on any other module, but a module in a subfolder shall not depend on a module at root. While we can hack this manually it feels fragile so better to find a way of enforcing it.

Some modules are potentially used by all others such as constants and maybe others. There could be a subset of modules put in one base or core subfolder and that all others depend on at least with respect to the CMakeLists.txt.

[jalvesz]

I just asked chatgpt to give some advice https://chatgpt.com/share/6960afff-a10c-800f-8493-948f753047d1 I think there are some nice ideas there!

One of the interesting points is about the src "leftovers" which it proposes to move them temporarily to a unsorted folder in the meantime and then progressively split it.

[jvdp1]

@jvdp1 after the issues encountered with #1086 it makes me think that we need to find a way of enforcing a strict hierarchy of dependencies to limit build time complexities.

I agree with you. One of the reasons we stopped supporting Makefile is that the depencies became to complex to be handle them easily. The modularization with Cmake leads to the same issues :(

The first level would be "a module at the root of src can depend on any other module, but a module in a subfolder shall not depend on a module at root. While we can hack this manually it feels fragile so better to find a way of enforcing it.

I am moving towards this goal in this PR. However, it is not so simple. Some linalg modules depend on e.g., a sort module that itself depends on another linalg module. Therefore creating subfolders based on the names only (e.g., stdlib_linalg, stdlib_sorting,...) is a too naive approach.

Some modules are potentially used by all others such as constants and maybe others. There could be a subset of modules put in one base or core subfolder and that all others depend on at least with respect to the CMakeLists.txt.

These will be easy to identify after that all other modules are moved to subdirectories. We can already identify 4-5 candidates for such a core subfolder. However, this should be done after all other modules are moved out of the src directory.

One of the interesting points is about the src "leftovers" which it proposes to move them temporarily to a unsorted folder in the meantime and then progressively split it.

It is actually a good idea. This will allow us to introduce a core and a unsorted lib in addition to the ones already mentioned in this PR (I have actually a few more in my local repo; I'll try to push them all).

[jvdp1]

@jvdp1 I see that the unsorted module basically contains: sparse matrices and the iterative solvers. I think these two can be nicely splitted in two: sparse and linalg_iterative

Done!

[jalvesz]

@jvdp1 I recovered your branch to test, I managed to build it as is. After reviewing the CMakeLists.txt, I just saw a couple of things:
For the constants folder, there is no need to add strings: target_link_libraries(constants PUBLIC core)
For the sparse folder, as of now the dependencies would be: target_link_libraries(sparse PUBLIC constants sorting) (no need for linalg)

With these two changes I managed to rebuild as well.

[jalvesz]

LGTM @jvdp1 I think this will open some interesting doors, like for instance being able to build stdlib with nvfortran by excluding a few bricks.

[jvdp1]

Thank you @jalvesz for the review.

I am currently playing with this branch. I just noted that the lib libfortran_stdlib.a created and installed with CMake only contains the file related with stdlib_version.fypp. It means that the user has to mention all other libraries for the linker. I think that it makes things difficult to handle with, e.g., Makefile, but pkg_config can help here.

Furthermore, the current structure of the package installed by CMake does not correspond to the config templates. Therefore, it results in failures because the paths to the directory include and lib are not correct.

[jvdp1]

Due to this new structure, the config file for pkg-congif had to be adapted (also partly related with #1084 mentioned by @fiolj ) as well as the structure of the directory of the installed package. I introduced the file config/export_pc.cmake to retrieve the list of libraries in the right order, needed for the pkg-config file.

Edit: @fiolj tested successfully this branch with pkg-config (see this comment)

[jalvesz]

@jvdp1 quick question, I built and installed locally this branch under windows. I see the following archive:

From where I understand that only certain "modules" got actually separated into their own folder while all others just got sent to fortran_stdlib. Is this the expected archive architecture? I was expecting to see each of the new "modules" as their own independent folder. At this point I guess it is not that relevant. I'm also wondering why test-drive is being archived with stdlib when it is supposed to be a dependency only for testing the library.

[jvdp1]

@jvdp1 quick question, I built and installed locally this branch under windows. I see the following archive: From where I understand that only certain "modules" got actually separated into their own folder while all others just got sent to fortran_stdlib. Is this the expected archive architecture?

Did you pull the last commit? Here is what I have on my linux machine (I am not familiar with Windows):

The structure in your figure (mod files in subdirectories) breaks the building process with cmake and pkg-config (see the issue #1084) because the include files are not correctly defined in the templates. It seems to be the case since #1033.
Collecting all mod files in the include directory fixes this issue and allows both cmake and pkg-config to build a project from an installed stdlib project. This is fixed by this change and this change

I was expecting to see each of the new "modules" as their own independent folder.

Ideally, it should be the case. However, the templates for cmake and pkg-config must be modified such that the flag -I include all relevant subdirectories. It might be too much for this PR IMHO.

I'm also wondering why test-drive is being archived with stdlib when it is supposed to be a dependency only for testing the library.

It was the case since #1033 if I am correct. Now it is archived next to it (but still in the main stdlib folder).

[jalvesz]

Sorry, I had the git bash in a different branch. You are right, all *mod files are directly under include and all the *.a and directly under lib.

Ideally, it should be the case. However, the templates for cmake and pkg-config must be modified such that the flag -I include all relevant subdirectories. It might be too much for this PR IMHO.

I think that as long as the packaging is consistent it is not an issue: either all plainly under the parent folder or all under their own subfolder. No need to change anything in this regard ;)

I'm also wondering why test-drive is being archived with stdlib when it is supposed to be a dependency only for testing the library.

It was the case since #1033 if I am correct. Now it is archived next to it (but still in the main stdlib folder).

I guess I didn't notice that change, but seems unnecessary to archive a testing dependency. (Not a blocking issue, just pointing that this might be considered for later)

[jalvesz]

I think this PR is not only ready but we should consider trying to get it merged before more PRs are stacked. It is a pivotal and required housekeeping of the internal architecture and packaging.

[jvdp1]

I think this PR is not only ready but we should consider trying to get it merged before more PRs are stacked. It is a pivotal and required housekeeping of the internal architecture and packaging.

Thank you @jalvesz . @perazz Are you also happy with this PR? If yes, I think we can merge it.

I think it will be good to create a new version (I can open a PR for that) after this PR as it will modify quite significantly the structure of stdlib

[perazz]

@jvdp1 the modularization looks absolutely good to me. I have a question about the CI: would it be possible to require stdlib to pass a check on the stdlib-fpm repository? Some of the recent updates seem to have broken fpm builds again:

https://github.com/fortran-lang/fpm/actions/runs/21221617807/job/61057589928?pr=1231

[jvdp1]

@jvdp1 the modularization looks absolutely good to me.

Thank you. Then I propose that we merge it.

I have a question about the CI: would it be possible to require stdlib to pass a check on the stdlib-fpm repository?

I can do it in another PR. However, fpm v0.12.0 is currently used for the code coverage. So, do you mean a more recent version?

Some of the recent updates seem to have broken fpm builds again:

Do you mean recent updates of stdlib or of fpm?

[jvdp1]

OK. I will merge this PR, and open a PR for preparing v0.8.0