Investigate memory footprint of HDBSCAN

[jinsolp]

Investigate the memory footprint of HDBSCAN with the goal of:

• Determining the largest dataset size that can be processed with the current implementation
• Identifying key memory bottlenecks and outlining actionable plans to reduce or eliminate them

[jinsolp]

Script for Calculating Peak Memory

type_to_bytes = {"char":1, "half": 2, "int": 4, "float": 4, "int64": 8, "uint64": 8}

CooIdxT = "int64"
OutIdxT = "int64"
ValueT = "float"
IdxT = "int64"
nnzT = "uint64"

def get_peak_mem_bytes_new(n_rows, min_samples):
    python_output = n_rows * (type_to_bytes[ValueT] * 3 + type_to_bytes[OutIdxT] * 4) \
        + (n_rows-1) * (type_to_bytes[ValueT] + type_to_bytes[OutIdxT] * 2)
    cpp_labels = n_rows * type_to_bytes[OutIdxT]
    bytes_per_nnz = 2 * type_to_bytes[CooIdxT] + type_to_bytes[ValueT]
    inds_and_dists = n_rows * min_samples * (type_to_bytes["float"] + type_to_bytes["int64"])
    mr_coo = n_rows * min_samples * 2 * bytes_per_nnz
    coo_rows = n_rows * min_samples * type_to_bytes[CooIdxT]
    symmetrize = n_rows * min_samples * 2 * bytes_per_nnz
    coo_sort = n_rows * min_samples * 2 * bytes_per_nnz
    return python_output + cpp_labels + mr_coo + inds_and_dists + coo_rows + symmetrize + coo_sort