When loading data from catalogs using the get_quantities method with return_iterator=True, it seems that the memory is not being freed as expected. The following simple example quickly overflows the memory (10GB by default on CC-IN2P3).
dc2 = GCRCatalogs.load_catalog('dc2_object_run2.2i_dr6')
for x in dc2.get_quantities(['ra', 'dec'], return_iterator=True):
pass
@yymao suggested explicitely closing handles by adding dc2.close_all_file_handles() in the loop, which does seem to correct the issue, although it is probably not the desired behaviour.
When loading data from catalogs using the
get_quantitiesmethod withreturn_iterator=True, it seems that the memory is not being freed as expected. The following simple example quickly overflows the memory (10GB by default on CC-IN2P3).@yymao suggested explicitely closing handles by adding
dc2.close_all_file_handles()in the loop, which does seem to correct the issue, although it is probably not the desired behaviour.