HDF5 sanity check failure: Sibling GID mirror inconsistency with AddressSanitizer enabled

[hsinhaoHHuang]

🔎 What happened?

We expect sibling->sibling_mirror = itself ([SibGID][ MirrorSib[s] ] == GID) for a patch.
For example, normally it should be this in the HDF5 snapshot:

['Tree']['Sibling'][4478][17] = 9840
['Tree']['Sibling'][9840][14] = 4478

where GID 4478 and GID 9840 are sibling patches (and 14 and 17 are MirrorSib mutually).

However, in some cases running with AddressSanitizer, the values are corrupted for unknown reasons, e.g.,

['Tree']['Sibling'][4478][17] = 9840
['Tree']['Sibling'][9840][14] = 0

If GAMER_DEBUG is enabled, there will be an error from the sanity check in Output/Output_DumpData_Total_HDF5.cpp:

Output_DumpData_Total_HDF5 (DumpID = 0)     ...
********************************************************************************
ERROR : Lv 1, GID 4478, sib 17, SibGID 9840 != SibGID->sibling 0 !!
        Rank <0>, file <Output/Output_DumpData_Total_HDF5.cpp>, line <1623>, function <Output_DumpData_Total_HDF5>
********************************************************************************

If AddressSanitizer is disabled, without any other changes, the values are normal.

📃 Steps to reproduce

1. Copy the ClusterMerger test problem from example/Hydro/ClusterMerger/*.
2. Build GAMER with the machine configuration file eureka_gnu.config without GPU and with the sanitizer flags -fsanitize=undefined -fsanitize=address enabled.
3. Run gamer until the first snapshot Data_000000 is output.
4. Check the sibling GID in the HDF5 snapshot

   python -c "import h5py; print( h5py.File('Data_000000', 'r')['Tree']['Sibling'][4478][17] )"
   python -c "import h5py; print( h5py.File('Data_000000', 'r')['Tree']['Sibling'][9840][14] )"
   

⌚ Commit hash

ce3bb85

🔧 Configuration command

python configure.py --machine=eureka_gnu --mpi=true --hdf5=true --fftw=FFTW3 --gpu=false --model=HYDRO --particle=true --gravity=true --passive=2 --par_attribute_int=1

🔨 Source files modified

configs/eureka_gnu.config

@@ -42,8 +42,8 @@ NVCCFLAG_POT -Xptxas -dlcm=ca
 # for debugging
 #CXXFLAG -fstack-protector-all
 #CXXFLAG -fstack-protector-strong
-#CXXFLAG -fsanitize=undefined -fsanitize=address
-#LIBFLAG -fsanitize=undefined -fsanitize=address
+CXXFLAG -fsanitize=undefined -fsanitize=address
+LIBFLAG -fsanitize=undefined -fsanitize=address
 
 # gpu
 GPU_COMPUTE_CAPABILITY 750    # 2080 Ti

💻 Operating system

linux (x86)

💾 Machine configuration file

eureka_gnu.config with sanitizer flags turned on:

CXXFLAG -fsanitize=undefined -fsanitize=address
LIBFLAG -fsanitize=undefined -fsanitize=address

🔖 Related topics

• Hydro
• MHD
• FDM
• AMR
• Gravity
• Particle
• Parallelization
• GPU
• Memory
• YT
• Tool
• Docs
• Other

💬 Additional information

1. Not sure whether it is related: during the compilation with ASan, it showed

Compiling LoadBalance/LB_RecordExchangeDataPatchID.cpp
LoadBalance/LB_RecordExchangeDataPatchID.cpp: In function ‘void LB_RecordExchangeDataPatchID(int, bool)’:
LoadBalance/LB_RecordExchangeDataPatchID.cpp:28:6: note: variable tracking size limit exceeded with ‘-fvar-tracking-assignments’, retrying without
   28 | void LB_RecordExchangeDataPatchID( const int Lv, const bool AfterRefine )
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compiling LoadBalance/LB_GetBufferData.cpp

2. Somehow this issue cannot be reproduced by the test problems BlastWave and Plummer.

3. We are inclined to believe there is an unexposed bug in GAMER rather than a problem in ASan so far.

[hsinhaoHHuang]

Another hint: When running without --mpi=false, the GID values are normal. Instead, there is another ASan runtime error:

Output_DumpData_Total_HDF5 (DumpID = 0)     ...
LoadBalance/LB_GatherTree.cpp:601:10: runtime error: index 26 out of bounds for type 'int [26]'
LoadBalance/LB_GatherTree.cpp:601:10: runtime error: index 26 out of bounds for type 'int [26]'
LoadBalance/LB_GatherTree.cpp:604:10: runtime error: index 3 out of bounds for type 'int [3]'
LoadBalance/LB_GatherTree.cpp:604:10: runtime error: index 3 out of bounds for type 'int [3]'
LoadBalance/LB_GatherTree.cpp:607:10: runtime error: index 3 out of bounds for type 'double [3]'
LoadBalance/LB_GatherTree.cpp:607:10: runtime error: index 3 out of bounds for type 'double [3]'
LoadBalance/LB_GatherTree.cpp:610:10: runtime error: index 3 out of bounds for type 'double [3]'
LoadBalance/LB_GatherTree.cpp:610:10: runtime error: index 3 out of bounds for type 'double [3]'
Output_DumpData_Total_HDF5 (DumpID = 0)     ... done

And another suspicion is that the supposedly contiguous memory allocation for the MPI Gatherv() may be changed by ASan's padding or alignment?

MPI_Gatherv( lel.SibList_Local[lv][0], amr->NPatchComma[lv][1]*26,      MPI_INT,
            (gel.SibList_AllLv+pc.GID_LvStart[lv])[0],        RecvCount_Sib,          RecvDisp_Sib,        MPI_INT,             root, MPI_COMM_WORLD );

[hsinhaoHHuang]

The runtime error when --mpi=false is due to the undefined behavior reported by UBSan.
This UB is related to how we treat a 2D array using pointers. We may expect the object (gel.SibList_AllLv+pc.GID_LvStart[lv])[0] to decay into a pointer to a 2D array, but it actually has a type of int [26]. Even though the memory for the array is contiguous, the compiler can assume the object will not be read beyond the bounds of 26. What we did here is to write a longer array of (26*number of patches) elements, which is a UB.
Here is a standalone C++ program to demonstrate this is UB.

#include <cstdio>
#include <typeinfo>

// Compile with UBSan: g++ -O1 -g -fsanitize=undefined ubsan_array2d_test.cpp -o ubsan_array2d_test

#define ROWS_OFFSET 1
#define ROWS 2
#define COLS 4

int main()
{

// allocate memory
   int (*array_1)[COLS] = new int[ROWS_OFFSET+ROWS][COLS];
   int (*array_2)[COLS] = new int[ROWS_OFFSET+ROWS][COLS];
   int (*array_3)[COLS] = new int[ROWS_OFFSET+ROWS][COLS];

/*------------------------------------------------------------------------------------------------------------------------*/
   printf("\n2D Way:            type = %s\n", typeid( array_1 ).name());

// assign
   const int offset_1 = ROWS_OFFSET;
   for (int r=0; r<ROWS; r++)
   for (int c=0; c<COLS; c++) { (array_1+offset_1)[r][c] = r*COLS + c; }

// print
   for (int r=ROWS_OFFSET; r<ROWS_OFFSET+ROWS; r++) { for (int c=0; c<COLS; c++) { printf(" %d", array_1[r][c]); } printf("\n"); }
/*------------------------------------------------------------------------------------------------------------------------*/
   printf("\nAcceptable 1D Way: type = %s\n", typeid( &array_2[0][0] ).name()); // 'int*'

// assign
   //const int offset_2 = ROWS_OFFSET * COLS;
   const int offset_2 = ROWS_OFFSET;
   for (int i=0; i<ROWS*COLS; i++) { (&(array_2+offset_2)[0][0])[i] = i; }

// print
   for (int r=ROWS_OFFSET; r<ROWS_OFFSET+ROWS; r++) { for (int c=0; c<COLS; c++) { printf(" %d", array_2[r][c]); } printf("\n"); }
/*------------------------------------------------------------------------------------------------------------------------*/
   printf("\nUB 1D Way:         type = %s\n", typeid( array_3[0] ).name());     // 'int [4]'

// assign
   const int offset_3 = ROWS_OFFSET;
   for (int i=0; i<ROWS*COLS; i++) {     ((array_3+offset_3)[0])[i] = i; }   // letting i run past COLS is **undefined behavior**.

// print
   for (int r=ROWS_OFFSET; r<ROWS_OFFSET+ROWS; r++) { for (int c=0; c<COLS; c++) { printf(" %d", array_3[r][c]); } printf("\n"); }
/*------------------------------------------------------------------------------------------------------------------------*/

// free memory
   delete [] array_1;
   delete [] array_2;
   delete [] array_3;

   return 0;
}

Although the result with the third method is still correct, the UBSan will complain this is UB.

2D Way:            type = PA4_i
 0 1 2 3
 4 5 6 7

Acceptable 1D Way: type = Pi
 0 1 2 3
 4 5 6 7

UB 1D Way:         type = A4_i
ubsan_array2d_test.cpp:43:67: runtime error: index 4 out of bounds for type 'int [4]'
 0 1 2 3
 4 5 6 7

For the code in GAMER, we can change it from lel.SibList_Local[lv][0] and (gel.SibList_AllLv+pc.GID_LvStart[lv])[0] to &lel.SibList_Local[lv][0][0] and &(gel.SibList_AllLv+pc.GID_LvStart[lv])[0][0] to avoid this UB.

[hsinhaoHHuang]

Even after fixing the above UB, the original issue when MPI is enabled is still not resolved.

The error may be related to MPI and unrelated to HDF5.
The occurrence of the wrong SibGID can be traced back to the MPI_Gatherv() inside LB_FillGlobalPatchExchangeList() in src/LoadBalance/LB_GatherTree.cpp.
Before the MPI_Gatherv(), the local array lel.SonList_Local[lv] of each MPI rank contains the correct sibling GID values. Right after the MPI_Gatherv(), some gathered values in the global array gel.SonList_AllLv incorrectly become zero.

By adding these checks

diff --git a/src/LoadBalance/LB_GatherTree.cpp b/src/LoadBalance/LB_GatherTree.cpp
index f74d8c4..6f28477 100644
--- a/src/LoadBalance/LB_GatherTree.cpp
+++ b/src/LoadBalance/LB_GatherTree.cpp
@@ -598,9 +598,51 @@ void LB_FillGlobalPatchExchangeList( LB_PatchCount& pc, LB_LocalPatchExchangeLis
          MPI_Gatherv( lel.SonList_Local[lv],    amr->NPatchComma[lv][1],         MPI_INT,
                       gel.SonList_AllLv+pc.GID_LvStart[lv],             RecvCount_Son,          RecvDisp_Son,        MPI_INT,             root, MPI_COMM_WORLD );
 
-         MPI_Gatherv( lel.SibList_Local[lv][0], amr->NPatchComma[lv][1]*26,      MPI_INT,
-                      (gel.SibList_AllLv+pc.GID_LvStart[lv])[0],        RecvCount_Sib,          RecvDisp_Sib,        MPI_INT,             root, MPI_COMM_WORLD );
-
+//-------------------------------------------------------------------------------------------------------------
+         int *SendBuff_Sib = ( amr->NPatchComma[lv][1] > 0 ) ? &lel.SibList_Local[lv][0][0] : NULL;
+         int *RecvBuff_Sib = ( amr->NPatchComma[lv][1] > 0  &&  MPI_Rank == root ) ? &(gel.SibList_AllLv+pc.GID_LvStart[lv])[0][0] : NULL;
+//-------------------------------------------------------------------------------------------------------------
+         if ( lv == 1 )
+         {
+            if ( MPI_Rank == 0 )
+               printf( "Before MPI_Gatherv, Rank-%d, Lv %d, GID %d, sib %d, SibGID %d\n",
+                       MPI_Rank, lv, 382+pc.GID_Offset[lv], 17, SendBuff_Sib[382*26+17] );
+            if ( MPI_Rank == 1 )
+               printf( "Before MPI_Gatherv, Rank-%d, Lv %d, GID %d, sib %d, SibGID %d\n",
+                       MPI_Rank, lv, 2352+pc.GID_Offset[lv], 14, SendBuff_Sib[2352*26+14] );
+         }
+//-------------------------------------------------------------------------------------------------------------
+         MPI_Gatherv( SendBuff_Sib,             amr->NPatchComma[lv][1]*26,      MPI_INT,
+                      RecvBuff_Sib,                                     RecvCount_Sib,          RecvDisp_Sib,        MPI_INT,             root, MPI_COMM_WORLD );
+//-------------------------------------------------------------------------------------------------------------
+         if ( MPI_Rank == root  &&  lv == 1 )
+         {
+            printf( "After MPI_Gatherv, Rank-%d, Lv %d, GID %d, sib %d, SibGID %d\n",
+                    MPI_Rank, lv,
+                    382+pc.GID_LvStart[lv],
+                    17,
+                    RecvBuff_Sib[382*26+17] );
+            printf( "After MPI_Gatherv, Rank-%d, Lv %d, GID %d, sib %d, SibGID %d\n",
+                    MPI_Rank, lv,
+                    pc.NPatchAllRank[1][1]+2352+pc.GID_LvStart[lv],
+                    14,
+                    RecvBuff_Sib[(pc.NPatchAllRank[1][1]+2352)*26+14] );
+
+            for (int GID=pc.GID_LvStart[lv]; GID<pc.GID_LvStart[lv]+NPatchTotal[lv]; GID++)
+            for (int s=0; s<26; s++)
+            {
+               const int SibGID = gel.SibList_AllLv[GID][s];
+
+               if ( SibGID >= 0 )
+               {
+                  const int MirrorSib[26] = { 1,0,3,2,5,4,9,8,7,6,13,12,11,10,17,16,15,14,25,24,23,22,21,20,19,18 };
+                  if ( gel.SibList_AllLv[SibGID][MirrorSib[s]] != GID )
+                     Aux_Error( ERROR_INFO, "Lv %d, GID %d, sib %d, SibGID %d != SibGID->sibling %d !!\n",
+                                lv, GID, s, SibGID, gel.SibList_AllLv[SibGID][MirrorSib[s] ] );
+               }
+            }
+         }
+//-------------------------------------------------------------------------------------------------------------
          MPI_Gatherv( lel.CrList_Local[lv][0],  amr->NPatchComma[lv][1]*3,       MPI_INT,
                       (gel.CrList_AllLv+pc.GID_LvStart[lv])[0],         RecvCount_Cr,           RecvDisp_Cr,         MPI_INT,             root, MPI_COMM_WORLD );

and run with 2 MPI ranks, we can get

Before MPI_Gatherv, Rank-1, Lv 1, GID 9840, sib 14, SibGID 4478
Before MPI_Gatherv, Rank-0, Lv 1, GID 4478, sib 17, SibGID 9840
After MPI_Gatherv, Rank-0, Lv 1, GID 4478, sib 17, SibGID 9840
After MPI_Gatherv, Rank-0, Lv 1, GID 9840, sib 14, SibGID 0
********************************************************************************
ERROR : Lv 1, GID 4478, sib 17, SibGID 9840 != SibGID->sibling 0 !!
        Rank <0>, file <LoadBalance/LB_GatherTree.cpp>, line <640>, function <LB_FillGlobalPatchExchangeList>
********************************************************************************

In fact, the SibGID = 0 happens for all the GID >= 9184

SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 0, SibGID 9181
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 1, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 2, SibGID 9182
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 3, SibGID 9222
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 4, SibGID 9180
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 5, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 6, SibGID 9179
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 7, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 8, SibGID 9219
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 9, SibGID 9195
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 10, SibGID 9177
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 11, SibGID 9217
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 12, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 13, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 14, SibGID 9178
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 15, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 16, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 17, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 18, SibGID 9176
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 19, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 20, SibGID 9216
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 21, SibGID 9192
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 22, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 23, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 24, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 25, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 0, SibGID 9969
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 1, SibGID 9185
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 2, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 3, SibGID 9186
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 4, SibGID 9307
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 5, SibGID 9187
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 6, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 7, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 8, SibGID 9972
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 9, SibGID 9188
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 10, SibGID 9333
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 11, SibGID 9309
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 12, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 13, SibGID 9189
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 14, SibGID 9990
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 15, SibGID 9974
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 16, SibGID 9310
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 17, SibGID 9190
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 18, SibGID 9983
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 19, SibGID 9335
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 20, SibGID 9991
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 21, SibGID 9311
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 22, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 23, SibGID -1
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 24, SibGID 9975
SendBuff Before MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 25, SibGID 9191

RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 0, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 1, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 2, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 3, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 4, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 5, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 6, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 7, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 8, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 9, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 10, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 11, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 12, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 13, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 14, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 15, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 16, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 17, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 18, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 19, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 20, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 21, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 22, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 23, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 24, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 25, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 0, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 1, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 2, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 3, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 4, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 5, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 6, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 7, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 8, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 9, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 10, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 11, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 12, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 13, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 14, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 15, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 16, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 17, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 18, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 19, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 20, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 21, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 22, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 23, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 24, SibGID 0
RecvBuff Before MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 25, SibGID 0

Rank-0, MPI returns = 0
Rank-1, MPI returns = 0

SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 0, SibGID 9181
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 1, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 2, SibGID 9182
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 3, SibGID 9222
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 4, SibGID 9180
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 5, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 6, SibGID 9179
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 7, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 8, SibGID 9219
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 9, SibGID 9195
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 10, SibGID 9177
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 11, SibGID 9217
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 12, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 13, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 14, SibGID 9178
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 15, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 16, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 17, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 18, SibGID 9176
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 19, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 20, SibGID 9216
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 21, SibGID 9192
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 22, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 23, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 24, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9183, sib 25, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 0, SibGID 9969
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 1, SibGID 9185
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 2, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 3, SibGID 9186
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 4, SibGID 9307
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 5, SibGID 9187
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 6, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 7, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 8, SibGID 9972
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 9, SibGID 9188
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 10, SibGID 9333
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 11, SibGID 9309
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 12, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 13, SibGID 9189
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 14, SibGID 9990
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 15, SibGID 9974
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 16, SibGID 9310
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 17, SibGID 9190
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 18, SibGID 9983
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 19, SibGID 9335
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 20, SibGID 9991
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 21, SibGID 9311
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 22, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 23, SibGID -1
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 24, SibGID 9975
SendBuff After MPI_Gatherv, Rank-1, Lv 1, GID 9184, sib 25, SibGID 9191

RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 0, SibGID 9181
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 1, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 2, SibGID 9182
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 3, SibGID 9222
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 4, SibGID 9180
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 5, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 6, SibGID 9179
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 7, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 8, SibGID 9219
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 9, SibGID 9195
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 10, SibGID 9177
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 11, SibGID 9217
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 12, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 13, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 14, SibGID 9178
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 15, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 16, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 17, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 18, SibGID 9176
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 19, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 20, SibGID 9216
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 21, SibGID 9192
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 22, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 23, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 24, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9183, sib 25, SibGID -1
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 0, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 1, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 2, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 3, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 4, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 5, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 6, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 7, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 8, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 9, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 10, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 11, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 12, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 13, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 14, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 15, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 16, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 17, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 18, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 19, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 20, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 21, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 22, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 23, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 24, SibGID 0
RecvBuff After MPI_Gatherv, Rank-0, Lv 1, GID 9184, sib 25, SibGID 0

That is, some of the values are successfully transfered from Rank-1 and some are not.

The logic, array size, pointer offset, sendcount, recvcounts, and displs of this MPI function call are all checked, and nothing wrong is found. The MPI_Gatherv() with the same size of sending and receiving arrays can work in a standalone program.

Currently, we suspect some still unexposed UB elsewhere in GAMER is the culprit. Or, somehow the MPI library is really incompatible with ASan.

Some other info:

Rank-0, amr->NPatchComma[0][ 1] = 2048
Rank-1, amr->NPatchComma[0][ 1] = 2048
Rank-0, amr->NPatchComma[1][ 1] = 3392
Rank-1, amr->NPatchComma[1][ 1] = 3392

Rank-0, pc.NPatchLocal[0] = 2048
Rank-1, pc.NPatchLocal[0] = 2048
Rank-0, pc.NPatchLocal[1] = 3392
Rank-1, pc.NPatchLocal[1] = 3392

Rank-0, NPatchTotal[0] = 4096
Rank-1, NPatchTotal[0] = 4096
Rank-0, NPatchTotal[1] = 6784
Rank-1, NPatchTotal[1] = 6784

Rank-0, pc.NPatchAllRank[0][0] = 2048
Rank-0, pc.NPatchAllRank[1][0] = 2048
Rank-1, pc.NPatchAllRank[0][1] = 3392
Rank-1, pc.NPatchAllRank[1][1] = 3392

Rank-0, pc.GID_LvStart[0] = 0
Rank-1, pc.GID_LvStart[0] = 0
Rank-0, pc.GID_LvStart[1] = 4096
Rank-1, pc.GID_LvStart[1] = 4096
Rank-0, pc.GID_LvStart[2] = 10880
Rank-1, pc.GID_LvStart[2] = 10880

Rank-0, pc.GID_Offset[0] = 0
Rank-1, pc.GID_Offset[0] = 2048
Rank-0, pc.GID_Offset[1] = 4096
Rank-1, pc.GID_Offset[1] = 7488
Rank-0, pc.GID_Offset[2] = 10880
Rank-1, pc.GID_Offset[2] = 15008

Rank-0, pc.NPatchLocalAllLv = 13664
Rank-1, pc.NPatchLocalAllLv = 13672
Rank-0, pc.NPatchAllLv = 27336
Rank-1, pc.NPatchAllLv = 27336

Lv 0, RecvDisp_Fa [0] = 0
Lv 0, RecvCount_Fa[0] = 2048
Lv 0, RecvDisp_Sib [0] = 0
Lv 0, RecvCount_Sib[0] = 53248

Lv 0, RecvDisp_Fa [1] = 2048
Lv 0, RecvCount_Fa[1] = 2048
Lv 0, RecvDisp_Sib [1] = 53248
Lv 0, RecvCount_Sib[1] = 53248

Lv 1, RecvDisp_Fa [0] = 0
Lv 1, RecvCount_Fa[0] = 3392
Lv 1, RecvDisp_Sib [0] = 0
Lv 1, RecvCount_Sib[0] = 88192

Lv 1, RecvDisp_Fa [1] = 3392
Lv 1, RecvCount_Fa[1] = 3392
Lv 1, RecvDisp_Sib [1] = 88192
Lv 1, RecvCount_Sib[1] = 88192.