Skip hash table size estimation if already done #450
Open
kim-fehl wants to merge 2 commits intoDerrickWood:masterfrom
Open
Skip hash table size estimation if already done #450kim-fehl wants to merge 2 commits intoDerrickWood:masterfrom
kim-fehl wants to merge 2 commits intoDerrickWood:masterfrom
Conversation
Save estimated size into the file required_capacity.txt
The estimated hash table size ($required_capacity variable) is now stored in file `required_capacity.txt`. If estimation was done at the machine with insufficient RAM, the user can now restart building the database skipping the estimation step (which can take hours alone if the data order is hundreds of Gb)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The estimated hash table size ($required_capacity variable) is now stored in file
required_capacity.txt.If estimation was done at the machine with insufficient RAM, the user can now restart building the database skipping the estimation step (which can take hours alone if the data order is hundreds of Gb)
find library/added/ -name 'prelim_map_*.txt' | xargs cat > library/added/prelim_map.txthas been moved under if statement (if file exists, this step is not nessecary)In the future probably we should add some '--force' option so user can rebuild the database from scratch if they want to without manually deleting files
required_capacity.txtandseqid2taxid.map(in case of adding some files tolibrary/addedfolder, for instance).The changes were tested with a small Archaea subset of standard kraken2-RefSeq database.