Version 0.9.4#123
Merged
Merged
Conversation
and not point to a specific env
Merging Fix 3 changes into Main clone
Fixed rule kma,emm_typing and most finders Fixed E.coli yaml
Modified the configuration E.Coli config file for the assembly lineage rule Modified the rule for assembly lineages using "shell" rather than "run" and executing the convert_external_genome.py Modified the convert_external_genome.py to be executed as a script
Part of requirements for resolving #5
Added a pulbic E. coli sample as well as a Actinobacillus pleuropneumoniae sample to test the E. coli pipe as well as the pipe for unspecified samples. Addressess #6
DAMN dat examplesetgit add examples/! Relatives now in sample sheets...
Added a single extra BP at the end of the very first contigs in both samples. The BP is a duplicate of the previous last BP. #6
Added the resources folder to gitignore Minor fixes in workflow/
* Added AMRFinderPlus Added AMRfinderplus, requires preloaded DB pointed out by the Species_config file * Removed message, polished comments and added organisms support Addressed issues with pull request #8. Organism support is a bit yanky now though, as you would be expected to provide the --organism XYZ directly into the species specific config file. Alternately you would have to leave a blank string ('')
* Added rules for automatically building CGEFinders databases. Added logging to CGEfinder rules Databases for CGEfinders are now automatically downloaded and indexed with KMA's if their db folder is missing. Logging has been set up to enhance debugging and usage information. This commit contributes to #5 #6 and #7 * Prepared for automatic AMRFinderDB Premerge automatic DB of AMRFinder prior to implementation * Added AMRfinder DB and polished messaging With the addition of AMRfinderPlus, Automatic Database handling has to come along. Now the database is automatically installed if missing. In addition messages are now named after their rule names, to enhance debugging * Corrected log positions, Amended AMRFinder db setup to names Logs are now placed into subcategory folders (based on sample or according to subject e.g. Databases
…tion (#10) Some tools are not used in practise any more and are thus redundant. Objects related to KMERFinder & Resistance Genes detection rules have been purged. Co-authored-by: SimoneScrima <simonescrima@gmail.com>
* Added MLST By Torsten Seeman MLST has been added and formatted to fit with the Logging conventions and Messaging convensions of #9 * Ammend: Forgot comma in Snakefile --------- Co-authored-by: SimoneScrima <simonescrima@gmail.com>
…ed kmeraligner rule (#12) * Added temporary Automatic DB handler for Ecoli kmerAligner and polished kmeraligner rule Current Ecoli kmeraligner database is hosted as part of a repo, to simplify, only the db file is downloaded using wget (thus wget has been added to envs/kmeraligner.yaml). The repo is prob not stable (not maintained) and thus this fix should be seen as a temporary addition to #7. To make rule comply with latest Logging and Messaging setup, the kmeraligner rule have recieved a polishing. * Amend: Replaced wget with Curl Since firewalls seems to block wget, Curl is used as a replacement to wget.
AMRfinder_update results in an error if database for some reason is not completely setup. Thus, everytime the setup rule is executed, it MUST overwrite the preexisting (if it's the same version) to mitigate future setup errors. Also corrected EcoliKmerAligner message.
While these tasks are NOT discarded on the long run, excluding these makes code more readable, while retaining the enhancements in commit history.
Make sure to update the supported tools section in all future pull requests, when relevant
Updated README - Pull FIRST
SerotypeFinder database is now generated automatically (if missing). Added messaging and logging for serotypefinder rule.
* Added update rule for MLST Due to some limitations in MLST setup, database updates can't be tied to snakemake rules output, and thus made as a dependency of the MLST execution. To manually force update (TAKES TIME\!) run * Update db_setups.smk Added time warning on the update_MLST rule * Added database creation file The database creation file is now chained into the MLST run, ensuring that if the database creation file does not exist, the DB is build * Removed multithreading since resulted in error in dowloading the database --------- Co-authored-by: SimoneScrima <simonescrima@gmail.com>
The file is a simple way of investegating the creation date for each used databases in the pipe.
Routine only
* Update README.md * Changed the order of sections in the README.md * Added description of which files to change when including species * Update README.md * Removed abstract and TCO from TOC --------- Co-authored-by: Kasper Thystrup Karstensen <kathka@gmail.com>
* Strict minimization - Deletions Removed a lot of files relating to NBDev, to enhance the overview of the repository. The removed files are not used in the later development of this pipe. * Removed additional samplesheet
* Added documentation using materials for mkdocs * github actions * Renamed home to index to fix github actions * fixing github actions * fixes github actions in ci.yaml * Fixes ci.yaml * Fix mispell ci.yaml * Revamp of docs organization * Overhaul of the docs * Fixes ci.yaml * Fixes to homepage * Rework on the documentation and finally a logo * Updated Docs
* Added documentation using materials for mkdocs * Deployed github actions * Renamed home to index to fix github actions * Coneected github actions in ci.yaml * Added a logo --------- Authored-by: SimoneScrima <simonescrima@gmail.com>
* Added a src/ folder to uphold standards * Migrated helper_functions to an internal utils/ folder, potentially split into several themed files * Renamed variables based on agreed rules: UPPERCASE -> Constants (mostly for pkg installation location), lowercase -> Dynamic (samplesheet, outdir ... etc)
* Separated common used functions into aptly named scripts in the utils/ folder * Made utils a callable module by adding an __init__ * Formulated standard imports from utils and all file paths as all import object * Renamed a few variables * Create.py works, setup.py unfinished mmaseq.py unstable
After module creation mmaseq.py was a bit messy and usntable. Now fixed
3 modules have now been implemented: * mmadeploy: - Utilized to deploy conda environments and databases. Will download and examine a test dataset (option to run on minimalistic 1 sample only dataset) * mmacreate: - Utilized to create a samplesheet. The input directiory will be screened recursively for .fasta and paried end .fastq.gz files, which in turn will be written to output directory/samplesheet.tsv * mmaseq: - Will read the specified samplesheet, create a pipeline config and finally execute the pipeline utils/__init__py functionality removed and module functionality restored to each module script file to enhance code transparency
Kleborate requires AMRfinderplus for certain annotation tasks. Since both databases and environments are deployed separately, system path to AMRfinder DB doesn't exists in Kleborate. Solution: Created kleborate_amrfinder setup rule, which symlinks amrfinder DB into environment system folder. An issue has been created, requesting options to specify database paths: klebgenomics/Kleborate#111 Dynamic output file names can be a bit of a pain. For now output are expected as a directory and versioning have been removed. This means that final output will NOT be part of the longtable. Issues have been created internally: #120 and a request for a static output file name has been reported klebgenomics/Kleborate#110
*Added some stablity fixes when running from deploy * deploy no longer uses outdir, instead it writes to MMAseqTest in cur dir * if deploy --small is selected assemblies are NOT ignored for the sake of included at least one assembly pipeline step
Reorganization and Launch modularization
Deploy creates output into deploydir rather than cwd
* Added a TRACE level below DEBUG, to showcase minute details * Provided more details on the different modules in the description texts
* Streamlined logging functionality * Minor code fixes
* Added a TRACE level below DEBUG, to showcase minute details * Provided more details on the different modules in the description texts
…download * Connection are estbalished per host, based on ftp paths in reads.url * Missing test sample files are downloaded as chunks * Old chunks are removed if detected, as they indicate download errors
* Retries occurs per path, not per host * Downloading occurs outside try: connect * ftp dicsonnect migrated as individual function
* Changed the deploy:'--small' argument to deploy:'--update' to highlight that all rules will be run with this option. * Added a mmaseq:'--force' option which force reruns all nescesary rules. * Running mmadeploy --update ... will now ensure that databases are allways rebuild. To save time, the small dtaset will be used.
* Assembly prioirty is set to highest (2) while multiprocess anaylsis are set to medium (1) to ensure earliest possible assembly and multicores processing * Cores reserved for multicore rules set to Cores - 1 Rounded down to an even number. Meaning that using odd numbered cores is no longer suboptimal
Basic skeleton for building the package
Prepare to push to TestPypi
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Main