Skip to content

Memmap version improvements #8

@rcarson3

Description

@rcarson3

Created a version of the parser that makes use of a memmap for the underlying file read. It currently results in a small 1-3% performance win. However, I believe by simplifying the parse_txt mmap version we could see a larger win. This would be by for example by eliminating the branching in the main loop where we have to check if we're on the 1st line or not and if so we need to figure out how many columns are being provided.

Next, I think it might be worth checking to see if we can't create parallel version of things for the memmap version, and if we end up seeing any sort of performance wins for large files. I have an idea for how to do this that should work pretty well, but it'll require some work to get there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions