Laghos status

[pearce8]

Components

1. Source code, Build system: Make. Optimization for linear: Add element batching capabilities to 3D MassIntegrator mfem/mfem#5299. RAJA Bounds Raja launchbounds mfem/mfem#5349. Update CMake HIP build CEED/Laghos#215. MFEM release in August
2. Source code released @rfhaque adding mesh parameters Add options to generate mesh from command line CEED/Laghos#197 shared device umpire pool CEED/Laghos#198 Adding input option for umpire device pool size tomstitt/Laghos#1
3. Spack package.py: @rfhaque adding Umpire (merged Laghos spack-v.1 and umpire changes #1087 laghos changes #1138)
4. Ramble application.py laghos changes #1138
5. Benchpark experiment.py laghos changes #1138
6. +cuda experiment.py
7. +rocm experiment.py
8. Benchmark dependencies: MFEM, Hypre, Umpire, Metis
9. small test size problem for CI
10. Scaling (Weak, Strong, or both): look at performance, memory pools, etc. Laghos #689
11. Large scale problem size defined: 16M DOF for 3D linear, quadratic, cubic. -- @rfhaque see comment with details from Kenny below laghos throws an mfem error for non-default problem order inputs #1139
12. CI running on CPUs/CDash
13. CI running on NVIDIA GPUs/CDash
14. CI running on AMD GPUs/CDash
15. +single_memory experiment.py
16. +caliper experiment.py: @wdhawkins adding instrumentation Basic Caliper support CEED/Laghos#196
17. CPU-only performance analysis
18. CUDA performance analysis - on vector by April 1
19. ROCm performance analysis - weak scaling to 256 nodes of Tuo
20. Documentation (build, run instructions, performance demo). Andrew to update to EPM
21. Convergence criteria: Andrew to Add analytic solution for Laghos Sedov benchmarks#32 Add analytical Sedov shock solution CEED/Laghos#205 Andrew to write up validation.
22. laghos throws an mfem error for non-default problem order inputs #1139
23. laghos (mfem) fails on matrix at runtime #855
24. Hypre build configuration on AMG2023 and Laghos #1107
25. Laghos and Remhos: review device variable and n_resources #872
26. Laghos MFEM+RAJA #1291
27. Nondeterministic hangs at large scale
28. Magma instead of hipblas? Or is it only for dense matrices? Laghos should not be using blas type matrix calculations. Is hypre using it for communications? --- Not using blas at all.

Once #1136 is merged, we need to update all of the packages below accordingly (and upstream all of the ones in the repo if possible).

• https://github.com/spack/spack-packages/blob/develop/repos/spack_repo/builtin/packages/mfem/package.py
• https://github.com/LLNL/benchpark/blob/develop/repo/mfem/package.py
• https://github.com/spack/spack-packages/blob/develop/repos/spack_repo/builtin/packages/hypre/package.py
• https://github.com/LLNL/benchpark/blob/develop/repo/hypre/package.py
• https://github.com/spack/spack-packages/blob/develop/repos/spack_repo/builtin/packages/laghos/package.py
• https://github.com/LLNL/benchpark/blob/develop/repo/laghos/package.py

Validation

• High-resolution run which we want the vendors to project is set to run to 250 steps, which doesn't get very far in simulation time.
• It doesn't make sense to check correctness before simulation time t=0.3. Running the simulation at high resolution to that simulation time will take hours.
• Andrew ran lower resolution with correct and incorrect answers, the smallest resolution we see the error is 64x64x64
• Andrew will look at the runtimes for the lower resolution runs so we can think about what might be reasonable to ask
• Vlad Tomov wants to be included in the discussion we have with Rieben/Tzanio. He says Remhos has the same issue
• Olga will try to get Matt's take