How to run impactX accurately/ fast and reliable?

[OLuckyG]

Hello All,

Now, I will first like to say I love using ImpactX for space-charge effects and how nice and compact it is.

However, there seems to be lacking of examples on how to actually run impactx, at least I could not find good resources.

For python usage, it is amazing and very well rounded.

The question is how to determine the accurate physics and how to speed up the simulations while maintaining accuracy?

For grids less than 64x64x64, I find that a single MPI rank and a good amount of OMP threads seem to be working fastest for multigrid solver. However, when I try to use FFT solver, the specified OMP threads are not being used and it becomes a single core job mostly. If I use 48x48x48 for both multgrid and FFT solver, they seem to be fast enough.

One thing that is not mentioned is what to do with padding? For multigrid solver, the boundary condition is Dirichlet so the padding should be large enough so the fields dont get messed up, but for FFT solver it is open boundary conditions so the padding can be smaller? Here for padding usage, I am referring to prob_relative:

from impactx import Config, ImpactX, elements
sim = ImpactX()
sim.prob_relative = [2.2] 

But if the padding is too large, then number of cells per $\sigma$ of the beam becomes smaller so one will need larger amount of grid which leads to larger particle count, and slower simulation.

For a single MPI rank case, I am using the following OMP environment sets:

import os

os.environ["OMP_NUM_THREADS"] = "48"
os.environ["OMP_PLACES"] = "cores"
os.environ["OMP_PROC_BIND"] = "spread"
os.environ["OMP_DYNAMIC"] = "FALSE"

# Add these for stack robustness
os.environ["OMP_STACKSIZE"] = "256M"     # important for multi-thread stability
os.environ["KMP_STACKSIZE"] = "256M"     # helps if Intel OpenMP runtime is in play

os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"

# Optional but useful for “silent” native crashes:
os.environ["PYTHONFAULTHANDLER"] = "1"

I know people say to do convergence tests, but it is almost impossible for me to do a convergence test for FFT solver on 64x64x64 grid where the bottle neck is the number of FFT calls. I thought using MPI will speed things up, but it does not for my case at least.

Also, I am reading my distribution from a file, and using MPI just duplicates the number of particles on each rank instead of distributing them. So to accurately distribute particles, I need to read the distribution on rank 0, and resize the mesh, and than call redistribute function which I am not sure if this is mentioned anywhere on the docs before.

I guess my questions are:

1. How to determine the correct padding? For FFT solver and Multigrid solver. Examples on read.docs seem to be only on a single focusing channel/ or drifts and no examples regarding larger simulation scales with rings. Is there a specific standard for FFT solver or Multigrid solvers? In examples I see, for FFT 1.2-1.4, and for multigrid 3.0 prob_relative values are picked. But the grid in the examples are too low and number of macro particles are also low, so how does that work with accuracy? In terms of number of grid cells per $\sigma_{x,y,t}$...
2. If the padding needs to be computed from convergence tests, then how can one speed up FFT solver? Is it only for my case that I am getting slow results? When I check the tiny profilers for time spent:
   48x48x48 grid 800K macro particles, padding 1.5, 20 OMP threads, a single turn in ring.

TinyProfiler total time across processes [min...avg...max]: 35.76 ... 35.76 ... 35.76

--------------------------------------------------------------------------------------------------------------
Name                                                           NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
--------------------------------------------------------------------------------------------------------------
ParticleContainer::RedistributeCPU()                              400      10.41      10.41      10.41  29.13%
impactx::transformation::CoordinateTransformation                 800      5.154      5.154      5.154  14.41%
FFT::R2C::forward(in)                                             800      4.336      4.336      4.336  12.13%
impactx::spacecharge::GatherAndPush                               400      3.313      3.313      3.313   9.26%
ImpactXParticleContainer::DepositCharge                           400      3.205      3.205      3.205   8.96%
FFT::R2C::backward(out)                                           400      3.071      3.071      3.071   8.59%
OpenBCSolver::setGreensFunction                                   400      1.995      1.995      1.995   5.58%
impactx::push::CFbend                                             180     0.4391     0.4391     0.4391   1.23%
OpenBCSolver::solve                                               400     0.3178     0.3178     0.3178   0.89%
impactx::push::Drift                                               96     0.3102     0.3102     0.3102   0.87%
impactx::transformation::CoordinateTransformation::to_fixed_s      53     0.2912     0.2912     0.2912   0.81%
impactx::push::Sol                                                117      0.237      0.237      0.237   0.66%
ImpactXParticleContainer::MinAndMaxPositions                      402     0.2112     0.2112     0.2112   0.59%
FabArray::setVal()                                               2400      0.181      0.181      0.181   0.51%
impactX::collect_lost_particles                                   400     0.1793     0.1793     0.1793   0.50%
impactx::push::BeamMonitor                                          2     0.1783     0.1783     0.1783   0.50%
ablastr::particles::deposit_charge::ChargeDeposition               54     0.1737     0.1737     0.1737   0.49%
impactx::transformation::CoordinateTransformation::to_fixed_t      55     0.1538     0.1538     0.1538   0.43%
impactx::particles::wakefields::HandleSpacecharge                 400     0.1139     0.1139     0.1139   0.32%
FabArray::ParallelCopy_nowait()                                   800     0.1118     0.1118     0.1118   0.31%
impactx::spacecharge::ForceFromSelfFields                         400    0.07499    0.07499    0.07499   0.21%
ImpactX::evolve::slice_step                                       400    0.01385    0.01385    0.01385   0.04%
impactx::spacecharge::PoissonSolve                                400    0.01365    0.01365    0.01365   0.04%
ImpactX::ResizeMesh                                               401    0.01025    0.01025    0.01025   0.03%
ablastr::fields::computePhiIGF                                    400   0.009439   0.009439   0.009439   0.03%
impactx::push                                                     400   0.001023   0.001023   0.001023   0.00%
computePhi                                                        400  0.0008178  0.0008178  0.0008178   0.00%
FabArray::ParallelCopy()                                          800  0.0006794  0.0006794  0.0006794   0.00%
ImpactX::track_particles                                            1  0.0002856  0.0002856  0.0002856   0.00%
Other                                                            5683     0.1904     0.1904     0.1904   0.53%
--------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------
Name                                                           NCalls  Incl. Min  Incl. Avg  Incl. Max   Max %
--------------------------------------------------------------------------------------------------------------
ImpactX::track_particles                                            1       34.7       34.7       34.7  97.03%
ImpactX::evolve::slice_step                                       400      34.69      34.69      34.69  97.02%
impactx::particles::wakefields::HandleSpacecharge                 400      33.27      33.27      33.27  93.04%
ParticleContainer::RedistributeCPU()                              400      10.41      10.41      10.41  29.13%
impactx::spacecharge::PoissonSolve                                400      10.05      10.05      10.05  28.11%
computePhi                                                        400      9.978      9.978      9.978  27.91%
ablastr::fields::computePhiIGF                                    400      9.968      9.968      9.968  27.88%
OpenBCSolver::solve                                               400      5.766      5.766      5.766  16.13%
impactx::transformation::CoordinateTransformation                 800      5.599      5.599      5.599  15.66%
FFT::R2C::forward(in)                                             800      4.336      4.336      4.336  12.13%
OpenBCSolver::setGreensFunction                                   400      4.191      4.191      4.191  11.72%
ImpactXParticleContainer::DepositCharge                           400      3.452      3.452      3.452   9.65%
impactx::spacecharge::GatherAndPush                               400      3.313      3.313      3.313   9.26%
FFT::R2C::backward(out)                                           400      3.115      3.115      3.115   8.71%
impactx::push                                                     400      1.229      1.229      1.229   3.44%
impactx::push::CFbend                                             180     0.4392     0.4392     0.4392   1.23%
impactx::push::Drift                                               96     0.3103     0.3103     0.3103   0.87%
impactx::transformation::CoordinateTransformation::to_fixed_s      53     0.2912     0.2912     0.2912   0.81%
impactx::push::Sol                                                117      0.237      0.237      0.237   0.66%
ImpactX::ResizeMesh                                               401     0.2214     0.2214     0.2214   0.62%
impactx::push::BeamMonitor                                          2     0.2137     0.2137     0.2137   0.60%
ImpactXParticleContainer::MinAndMaxPositions                      402     0.2112     0.2112     0.2112   0.59%
FabArray::setVal()                                               2400      0.181      0.181      0.181   0.51%
impactX::collect_lost_particles                                   400     0.1793     0.1793     0.1793   0.50%
ablastr::particles::deposit_charge::ChargeDeposition               54     0.1737     0.1737     0.1737   0.49%
impactx::transformation::CoordinateTransformation::to_fixed_t      55     0.1538     0.1538     0.1538   0.43%
FabArray::ParallelCopy()                                          800     0.1131     0.1131     0.1131   0.32%
FabArray::ParallelCopy_nowait()                                   800     0.1124     0.1124     0.1124   0.31%
impactx::spacecharge::ForceFromSelfFields                         400     0.1038     0.1038     0.1038   0.29%
Other                                                            5683     0.3083     0.3083     0.3083   0.86%
--------------------------------------------------------------------------------------------------------------

Pinned Memory Usage:
--------------------------------------------------------------------
Name                                Nalloc  Nfree   AvgMem    MaxMem
--------------------------------------------------------------------
ParticleContainer::addParticles         72     72  309 KiB    55 MiB
The_Pinned_Arena::Initialize()           1      1  611   B  8192 KiB
ImpactX::early_param_check               1      1    0   B    32   B
ImpactX::init_grids                      2      2    0   B    32   B
ImpactX::track_particles                 2      2    0   B    32   B
impactx::push                            3      3    0   B    32   B
impactx::spacecharge::PoissonSolve    1596   1596    0   B    32   B
--------------------------------------------------------------------

Cpu Memory Usage:
--------------------------------------------------------------------------
Name                                     Nalloc  Nfree    AvgMem    MaxMem
--------------------------------------------------------------------------
FFT::R2C                                      2      2    19 MiB    20 MiB
ablastr::fields::computePhiIGF                1      1    10 MiB    10 MiB
ImpactX::init_grids                          40     40  7955 KiB  7959 KiB
FillBoundary_nowait()                     22400  22400   510   B   395 KiB
ImpactXParticleContainer::DepositCharge    1600   1600  4728   B   110 KiB
--------------------------------------------------------------------------

AMReX (25.12) finalized
Total execution time: 0.60 mins

When I use 64x64x64 with the same setup:

TinyProfiler total time across processes [min...avg...max]: 124.2 ... 124.2 ... 124.2

--------------------------------------------------------------------------------------------------------------
Name                                                           NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
--------------------------------------------------------------------------------------------------------------
FFT::R2C::forward(in)                                             800      69.16      69.16      69.16  55.69%
FFT::R2C::backward(out)                                           400      35.41      35.41      35.41  28.52%
ParticleContainer::RedistributeCPU()                              400      5.075      5.075      5.075   4.09%
OpenBCSolver::setGreensFunction                                   400      4.019      4.019      4.019   3.24%
impactx::transformation::CoordinateTransformation                 800      2.712      2.712      2.712   2.18%
impactx::spacecharge::GatherAndPush                               400      1.725      1.725      1.725   1.39%
ImpactXParticleContainer::DepositCharge                           400      1.424      1.424      1.424   1.15%
impactx::transformation::CoordinateTransformation::to_fixed_s     227      0.584      0.584      0.584   0.47%
OpenBCSolver::solve                                               400      0.401      0.401      0.401   0.32%
ablastr::particles::deposit_charge::ChargeDeposition              222     0.3487     0.3487     0.3487   0.28%
impactx::push::CFbend                                             180     0.2964     0.2964     0.2964   0.24%
impactx::transformation::CoordinateTransformation::to_fixed_t     245     0.2789     0.2789     0.2789   0.22%
FabArray::ParallelCopy_nowait()                                   800     0.2375     0.2375     0.2375   0.19%
impactx::push::BeamMonitor                                          2      0.237      0.237      0.237   0.19%
impactx::push::Sol                                                117     0.1899     0.1899     0.1899   0.15%
impactx::particles::wakefields::HandleSpacecharge                 400     0.1241     0.1241     0.1241   0.10%
ImpactX::evolve::slice_step                                       400    0.01626    0.01626    0.01626   0.01%
impactx::spacecharge::PoissonSolve                                400    0.01459    0.01459    0.01459   0.01%
ablastr::fields::computePhiIGF                                    400    0.01448    0.01448    0.01448   0.01%
FabArray::ParallelCopy()                                          800   0.001644   0.001644   0.001644   0.00%
impactx::push                                                     400  0.0009322  0.0009322  0.0009322   0.00%
computePhi                                                        400  0.0008181  0.0008181  0.0008181   0.00%
ImpactX::track_particles                                            1  0.0003462  0.0003462  0.0003462   0.00%
Other                                                            9950     0.9104     0.9104     0.9104   0.73%
--------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------
Name                                                           NCalls  Incl. Min  Incl. Avg  Incl. Max   Max %
--------------------------------------------------------------------------------------------------------------
ImpactX::track_particles                                            1      123.2      123.2      123.2  99.20%
ImpactX::evolve::slice_step                                       400      123.2      123.2      123.2  99.19%
impactx::particles::wakefields::HandleSpacecharge                 400        122        122        122  98.28%
impactx::spacecharge::PoissonSolve                                400      109.4      109.4      109.4  88.14%
computePhi                                                        400      109.4      109.4      109.4  88.07%
ablastr::fields::computePhiIGF                                    400      109.3      109.3      109.3  88.06%
OpenBCSolver::solve                                               400      70.66      70.66      70.66  56.90%
FFT::R2C::forward(in)                                             800      69.16      69.16      69.16  55.69%
OpenBCSolver::setGreensFunction                                   400      38.67      38.67      38.67  31.14%
FFT::R2C::backward(out)                                           400      35.48      35.48      35.48  28.57%
ParticleContainer::RedistributeCPU()                              400      5.075      5.075      5.075   4.09%
impactx::transformation::CoordinateTransformation                 800      3.575      3.575      3.575   2.88%
ImpactXParticleContainer::DepositCharge                           400      1.859      1.859      1.859   1.50%
impactx::spacecharge::GatherAndPush                               400      1.725      1.725      1.725   1.39%
impactx::push                                                     400     0.9263     0.9263     0.9263   0.75%
impactx::transformation::CoordinateTransformation::to_fixed_s     227      0.584      0.584      0.584   0.47%
ablastr::particles::deposit_charge::ChargeDeposition              222     0.3487     0.3487     0.3487   0.28%
impactx::push::CFbend                                             180     0.2965     0.2965     0.2965   0.24%
impactx::transformation::CoordinateTransformation::to_fixed_t     245     0.2789     0.2789     0.2789   0.22%
impactx::push::BeamMonitor                                          2     0.2619     0.2619     0.2619   0.21%
FabArray::ParallelCopy()                                          800     0.2402     0.2402     0.2402   0.19%
FabArray::ParallelCopy_nowait()                                   800     0.2386     0.2386     0.2386   0.19%
impactx::push::Sol                                                117       0.19       0.19       0.19   0.15%
Other                                                            9950      1.163      1.163      1.163   0.94%
--------------------------------------------------------------------------------------------------------------

Pinned Memory Usage:
--------------------------------------------------------------------
Name                                Nalloc  Nfree   AvgMem    MaxMem
--------------------------------------------------------------------
ParticleContainer::addParticles        171    171  114 KiB    55 MiB
The_Pinned_Arena::Initialize()           1      1  174   B  8192 KiB
ImpactX::early_param_check               1      1    0   B    32   B
ImpactX::init_grids                      2      2    0   B    32   B
ImpactX::track_particles                 2      2    0   B    32   B
impactx::push                            3      3    0   B    32   B
impactx::spacecharge::PoissonSolve    1596   1596    0   B    32   B
--------------------------------------------------------------------

Cpu Memory Usage:
-------------------------------------------------------------------------
Name                                     Nalloc  Nfree    AvgMem   MaxMem
-------------------------------------------------------------------------
FFT::R2C                                      2      2    43 MiB   43 MiB
ablastr::fields::computePhiIGF                1      1    21 MiB   22 MiB
ImpactX::init_grids                          40     40    15 MiB   15 MiB
FillBoundary_nowait()                     22400  22400   307   B  669 KiB
ImpactXParticleContainer::DepositCharge    4056   4056  1354   B  221 KiB
-------------------------------------------------------------------------

AMReX (25.12) finalized
Total execution time: 2.08 mins

For reference simulation properties:

sim.n_cell = [64,64,64]
sim.poisson_solver = "fft"
sim.prob_relative = [1.5] 
sim.particle_shape = 3 
sim.space_charge = "3D"
sim.slice_step_diagnostics = False

Therefore, I do not understand why on 64x64x64 the FFT solver slows down drastically...

I have 48 physical cores with SMT so 96 threads, but it seems like SMT does not add anything to the speed so I am using physical cores.

Any suggestions/ help will be appreciated!

[ax3l]

Hi @OLuckyG,

Thank you for the kind words, detailed report and tests!

Before we dig into the performance details, can you share how you installed ImpactX and its dependencies?
In particular, since you do CPU simulations, was your FFTW compiled with OpenMP support?

With regards to padding:

• Our IGF FFT solver uses open boundaries, so something like 1.1x padding (10% larger than the beam maximum extend) will do 👍 Do not pad more, it just makes your resolution worse.
• Our MLMG solver uses dirichlet boundary conditions, so you need to pad a lot depending on your use case, to ensure the SC fields have fallen sufficiently to zero at the boundary. Our suggestion starts with 3x padding.

For MLMG, your padding is part of the convergence test. For the FFT solver, it is not.

Examples:
Our examples are intentionally coarse, because we run them quickly and all the time in continuous testing on small machines with 2-4 cores. Our SC examples often have a comment, like, "use this particle number for converged results", which is often ~100x more than the example. Along the same lines, we use a very coarse grid and you need to increase it for realistic/complex beams and long-term tracking.

[OLuckyG]

Hello @ax3l

Thank you for replying!

Yeah I also guessed it similarly with the FFT and MLMG solver with padding idea! Thank you for clarifying that!

As to the installation of ImpactX, I have installed it two ways!

First Way: Using the link https://impactx.readthedocs.io/en/latest/install/users.html , following the conda installation method.

conda create -n impactx -c conda-forge impactx
conda activate impactx

Then I realized that this installation does not support MPI, but it supports OMP in terms of running the simulation.

Second Way: I have compiled it from scratch because I realized the MPI support was off, so I thought maybe I can get some speed if I compile it from scratch. To do this I used another conda environment and installed dependencies as listed in: https://impactx.readthedocs.io/en/latest/install/dependencies.html#install-dependencies , then I compiled it following:

git clone https://github.com/BLAST-ImpactX/impactx.git $HOME/src/impactx
cd $HOME/src/impactx

conda activate impactx-cpu-mpich-dev

# configure
cmake -S . -B build -G Ninja \
  -DImpactX_MPI=ON \
  -DImpactX_FFT=ON \
  -DImpactX_PYTHON=ON

cmake --build build -j 4
cmake --build build --target pip_install -j 4  

---

As to answer the question about FFTW OMP links, for the environment that I compiled it, I can run a few commands:
FIrst:

echo "$CONDA_PREFIX"
ls -1 $CONDA_PREFIX/lib | grep -E 'fftw3.*_omp|fftw3f.*_omp|fftw3l.*_omp' || echo "No *_omp FFTW libs found"

/home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev
libfftw3f_omp.so
libfftw3f_omp.so.3
libfftw3f_omp.so.3.6.10
libfftw3l_omp.so
libfftw3l_omp.so.3
libfftw3l_omp.so.3.6.10
libfftw3_omp.so
libfftw3_omp.so.3
libfftw3_omp.so.3.6.10

To check if these are actually linked to the simulation, I run:

 ldd /home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev/lib/python3.12/site-packages/impactx/impactx_pybind.cpython-312-x86_64-linux-gnu.so | egrep -i 'fftw|gomp|omp|iomp'
        libgomp.so.1 => /home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev/lib/libgomp.so.1 (0x00007d843599e000)
        libfftw3.so.3 => /home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev/lib/libfftw3.so.3 (0x00007d8434200000)
        libfftw3_omp.so.3 => /home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev/lib/libfftw3_omp.so.3 (0x00007d8435996000)
        libfftw3f.so.3 => /home/oluckyg/miniconda3/envs/impactx-cpu-mpich-dev/lib/libfftw3f.so.3 (0x00007d8432000000)

So I am guessing it looks good?

Another thing I have not tried was, setting FFTW threads as environment parameter, I do not know if it will help or not..

For reference, here is my lscpu for performance suggestions:

lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   96
  On-line CPU(s) list:    0-95
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 9454P 48-Core Processor
    CPU family:           25
    Model:                17
    Thread(s) per core:   2
    Core(s) per socket:   48
    Socket(s):            1
    Stepping:             1
    Frequency boost:      enabled
    CPU(s) scaling MHz:   74%
    CPU max MHz:          3810.7910
    CPU min MHz:          1500.0000
    BogoMIPS:             5500.21
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1
                          gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16
                           pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dn
                          owprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perf
                          mon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx
                          512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_
                          local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_
                          scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip 
                          pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flu
                          sh_l1d debug_swap
Virtualization features:  
  Virtualization:         AMD-V
Caches (sum of all):      
  L1d:                    1.5 MiB (48 instances)
  L1i:                    1.5 MiB (48 instances)
  L2:                     48 MiB (48 instances)
  L3:                     256 MiB (8 instances)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-95
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Mitigation; Safe RET
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

---

Let me know what do you think! Because I am not sure why FFT is slower on the 64x64x64 and beyond. I thought MPI will increase the speed but it does not, I am guessing its because there isnt much to do for other ranks so MPI overhead dominates.. But I thought if AMREX blocking factor is 32, then using 64x64x64 -> creates 8 boxes so maybe using 4-6 ranks will speed things up but it does not. Or maybe I am thinking wrongly in terms of using MPI and it actually needs a lot more grids and particle counts to speed things up...

So with the FFT work, upto 48x48x48 grid I see all 48 cores being used, maybe not upto 100% but up there. However, at 64x64x64 it seems like only a single core is 100% and the rest is like 17% or so.. I can also this behavior of I run turbostat command. Turbostat shows at 48x48x48 the frequency is up there in terms of GHz, but when I use 64x64x64 it drops drastically.

It seems like the Green functions are computed a lot which I was wondering if fixing the box statically might fix that or something? Because the dynamic evaluation might keep calling the Green function solves on the domain and the FFT computation calls are also a lot.

I guess if the computation time for FFT scales: $NxNyNz * log(NxNyNz)$ then compared with 48x48x48 and 64x64x64 grids , the 64x64x64 should have 2.55 more time spend?

[ax3l]

Thank you for the details!

We think we found the problem: our FFTW usage in AMReX does not yet use threading/OpenMP for the FFTs, even when available.
This is an oversight, we had another initial implementation for the IGF that was implemented in ABLASTR, where the FFT wrappers already use threading when available.

We opened issues in ImpactX and AMReX for tracking:

• FFT SC Solver not yet Threaded #1249
• AMReX FFT: FFTW w/ OpenMP Threads AMReX-Codes/amrex#4856

[ax3l]

It seems like the Green functions are computed a lot which I was wondering if fixing the box statically might fix that or something? Because the dynamic evaluation might keep calling the Green function solves on the domain and the FFT computation calls are also a lot.

Yes, one could cache it if we keep the box fully static. We have not yet explored this in detail. There is also potential to speed up the green's function calculation using SIMD on CPUs, with these WIP PRs (not yet functional):

• [WIP] IGF SIMD BLAST-WarpX/warpx#6147
• [WIP] OpenBC SIMD AMReX-Codes/amrex#4649

[OLuckyG]

Okay this makes sense now...

When I wrote a tiny c code and pulled the FFT libraries with OMP, the FFT thread number were set to OMP thread number so the FFT can be threaded absolutely! 😄

Okay cool! Its my second bug finding for ImpactX 😆 , one with the monitor backend: h5 file requiring 1000GB of RAM for processing, and now this yeayyyy 😆 .

So I guess in theory, I can just use 48x48x48 grid with FFT solver with padding 1.2 with 800K macro particles should still be a good result.. (hopefully)

---

Another question I have is, can we use SMT for speed ups? Currently I am binding to physical cores, 48 in my case but there is SMT option so 96 threads.. does this help one way or another with speed or not? At the moment I have not seen any gain going beyond the physical cores but I am not sure if there is an option to activate something in the code or not...

Secondly: If FFT is a single "core" operation, will that suggest using MPI can speed things up? So the process being spawned on multiple cores....

[OLuckyG]

Also one more thing that I am seeing: I am not sure if this is on the simulation end or my machine end...

When I run impactx with python, I typically run it as:

nohup python3 -u script.py > run.log &

Or something like that..

When the simulation goes 1K turns or sometimes 200 turns it just dies off... No errors in the log file no stack dump available it just dies off and stops...

I have no idea why that happens... For both MLMG solver and FFT solver I experienced the same thing...

[ax3l]

Another question I have is, can we use SMT for speed ups? Currently I am binding to physical cores, 48 in my case but there is SMT option so 96 threads.. does this help one way or another with speed or not? At the moment I have not seen any gain going beyond the physical cores but I am not sure if there is an option to activate something in the code or not...

You can technically use SMT but I have not seen big speedups in practice. Whenever I benchmark it with AMReX codes I see about 5% improvement on Intel CPUs...

The reason behind that is involved, but SMT in general shares many resources of your CPU pipeline.

[ax3l]

nohup python3 -u script.py > run.log &

That is a bit weird -- and without nohup it works normally @OLuckyG ?

What machine is this on?

Sounds to me like a system watchdog process terminates the costly process you started and detached from your session (via nohup).
You can try using screen or tmux instead, in case you try to keep a process running on a server or machine that you log off from.

[ax3l]

Okay cool! Its my second bug finding for ImpactX 😆 , one with the monitor backend: h5 file requiring 1000GB of RAM for processing, and now this yeayyyy 😆 .

Congrats!! 👏 ✨ Always happy to receive reproducible bug reports :)

Waiting for your hat trick now :D

[OLuckyG]

@ax3l Hi Axel! I am sorry I did not select the answer! I thought to wait till the multithreading is patched...

I kind of figured why it terminates but I do not know the reason behind it..

Generally, I was running my shells in my server with tmux terminal emulator and it seems like when I run the simulation within tmux and the tmux session was exited, the process will hit a signal kill... I cant really prove why but this is the behavior I observed :D

Thanks for helping out! Also I am wondering when will the multithreaded FFT will come to the next patch?

Thanks!