Skip to content

[ENH] Add support for Memory Mapped Arrays in the MarkovChain Class #11

Description

@aadya940

chainopy was built for efficiently working with large transition-matrics and additionally reduce the drawbacks of the python langauge. However, for a corpus of words that have large number of unique words, the transition matrix scales up pretty quickly which doesn't have much workarounds. Hence, add support for numpy.memmap to work with memory mapped arrays.

Additional References: https://numpy.org/doc/stable/reference/generated/numpy.memmap.html

Rough Steps:

  • Modify the constructor accordingly.
  • Check if all methods are compatible with memmap.

Potential Benefits:

  • Large Datasets
  • Reduced Memory Footprint
  • Handling Large Markov Chains
  • Efficient Disk I/O
  • Data Persistence
  • Lazy Loading
  • Reduced Swap Usage

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions