cp: fix: async_utils: explicit GC in persistent checkpoint worker loop (3591) into core_r0.16.0 (#3628)

Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Seonmyeong Bak <sbak@nvidia.com>

Author: ko3n1g

megatron/core/dist_checkpointing/strategies/async_utils.py

@@ -505,6 +505,9 @@ def async_loop(
                 logger.debug(f"{rank} has completed saving {item.call_idx}")
                 comp_q.put(item.call_idx)
                 queue.task_done()
+                del async_fn_args
+            del item
+            gc.collect()
 
         logger.debug(f"PersistentAsyncCaller: persistent ckpt worker for {rank}  has terminated")