chainopy was built for efficiently working with large transition-matrics and additionally reduce the drawbacks of the python langauge. However, for a corpus of words that have large number of unique words, the transition matrix scales up pretty quickly which doesn't have much workarounds. Hence, add support for numpy.memmap to work with memory mapped arrays.
Additional References: https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
Rough Steps:
- Modify the constructor accordingly.
- Check if all methods are compatible with
memmap.
Potential Benefits:
- Large Datasets
- Reduced Memory Footprint
- Handling Large Markov Chains
- Efficient Disk I/O
- Data Persistence
- Lazy Loading
- Reduced Swap Usage
chainopywas built for efficiently working with large transition-matrics and additionally reduce the drawbacks of the python langauge. However, for a corpus of words that have large number of unique words, the transition matrix scales up pretty quickly which doesn't have much workarounds. Hence, add support fornumpy.memmapto work with memory mapped arrays.Additional References: https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
Rough Steps:
memmap.Potential Benefits: