Profiling the memory usage

[yuanzhou]

Based on observations from #943, over time (several hours to a day) the entity-api container (may also apply to search-api container and others) gradually consumes more RAM. Eventually, it may max out system resources unless restarting the container. This may have been contributing to the VM downtime issue as well.

Try with different Python memory profilers to analyze and understand the memory usage of entity-api. The goal is to identify memory leaks, optimize memory consumption, and improve the overall efficiency.

Combining the following activities:

• Create new entities to trigger reindex
• Update entities to trigger reindex
• Retrieve entities of different types

[yuanzhou]

It makes no sense with such high memory consumption while entity-api is not under load.

[yuanzhou]

10/28/2025, triggered full reindex on PROD and noticed entity-api memory usage is very high over 18G. Restarted the container and it went down to less than 300MB initially with no activities then up to 4G as the full reindexed triggered again.