add SINA reference based alignment

[epruesse]

TODO

• add tests
• expose parameters useful in Q2 context

Addition Description

Allow using SINA to align to reference (i.e. GreenGenes or SILVA)

Current Behavior

Only de-novo alignment using MaFFT

Proposed Behavior

Reference based alignment is useful for placement in pre-existing tree as alignments must match and promises better scalability than progressive de-novo alignment.

Comments

SINA (currently) requires ARB which in turn requires a ton of software, including X11, creating a bit of a problem for a core plugin.

@wasade told me to PR to qiime2/q2-alignment anyway for now. For production, the issue will need some kind of fix. Options that come to mind:

1. Making SINA a plugin of its own: As qiime alignment sina would be the natural spot, this would require a (clean) way to inject actions into core plugins.
2. Creating a Qiime2 specific version of SINA that works without ARB. This looses some functionality, however.

References
See PR #54

[wasade]

Thanks, @epruesse!

Another option would be, for now, do create a separate plugin as q2-sina which does pull down ARB, and to migrate the code to q2-alignment in the future once kinks have been worked out?

[epruesse]

@wasade If I went the q2-sina route, where would I host that package? It should be a conda package due to the binary dependencies, but would fail tests at Bioconda due to missing qiime2 dependencies (question came up here for ghost-tree as well).

I think a minimal feature route withq2-alignment using a sina-noarb variant package that is independent of ARB should be possible within a 2-week timeframe.

The primary features offered by the ARB dependency would not easily integrate with Qiime2 anyway. The ARB files are something of QZAs themselves, combining multiple sequence features, alignments, phylogenetic trees, taxonomic groupings, per-organism and per-alignment-column metadata, etc into one database file.

[wasade]

It's similar to the other plugins for hosting. See q2-fragment-insertion for example.

I think an ARB independent version of SINA would reduce deployment burden, and perhaps it would make sense as a PR on this repo.

@gregcaporaso or @thermokarst, any issues with a PR here vs. a separate q2 plugin?

[gregcaporaso]

I think including in q2-alignment would be a good way to go if possible. This would be a really nice addition - thanks for your interest in contributing it @epruesse!

[epruesse]

@gregcaporaso I had waited for a while, hoping someone might add PyNAST so that I could copy-paste more easily ;-)

[epruesse]

@wasade Is there a way to have software required by q2 plugins installed into separate conda environments? I believe we touched upon this topic at ISME.

Turns out that conda install sina in a qiime2-dev environment fails with

UnsatisfiableError: The following specifications were found to be in conflict:
  - freetype=2.9.1
  - q2-dada2=2018.8.0.dev0
  - sina

In this case I know the culprit - the openmotif package needs an update. Thinking long-term maintenance though, these things tend to become an endless rabbit hole. I'm not sure I will be able to guarantee that sina and its dependencies will remain conflict free with all other packages required by qiime2 base (in particular since sina uses boost...).

[thermokarst]

This is a good reason for moving away from the "core distribution" model and into an a la carte "choose your favorite plugins" style (@ebolyen and I have been discussing this at length in recent months). Then if there are conflicting plugins, we at least stand a chance (just make two different envs and boumce between them).

[epruesse]

OFF TOPIC - re dependency hell
@thermokarst @ebolyen A point for your discussion: While you obviously can't separate out Python dependencies that need to work in the same interpreter instance as qiime itself, you could create separate "per tool" conda environments for binary dependencies. I was thinking about that for SINA, but framework support would make things a lot easier.

Roughly, what I'd envision is defining an action with an extra parameter:

register_function(conda_env={'mycommand': {'bioconda': ['fastspar']}}, ...)

The function would then be passed an accessor object via which commands can be executed.

def myfunction(input1: type1, conda_env: qiime2.envrunner):
  conda_env.run("mycommand", ['mycommand', '-i', str(input1)])
  ...

Upon call, the accessor object can ensure that the environment is available, installing it on demand in a suitable place (same disk as conda installation to allow hard links to work), and prefix the command with source activate $PATH_TO_ENV;. The accessor could also handle verbosity/debugging such as implemented byq2_alignment.mafft.run_command in a more frameworky way.

This would limit dependency hell to the things needed by the plugin python code, which is much more controllable than the needs of the wrapped tools. Initial installation size would also be reduced, potentially allowing many more plugins to be delivered with qiime core.

The CI system could then actually pre-resolve the environments, run unit-tests and store fully qualified URLs to be used for creating environments on the user side. The latter would prevent user/support pain with changing packages in the conda channels, and also has the potential of greatly increasing installation speed.

I've got code doing essentially this for my own pipeline. If you want to derive from that I'll happily grant you license (code is GPL but all mine) to do so.

[epruesse]

Ok, I've got the dependency issue fixed. conda install -c bioconda sina now requires only arb-bio-tools, boost, and libarbdb in addition to sina itself. Total installed size is ~200MB, of which most is boost (~180MB).

How do I handle the ARB files and the indices that SINA needs for alignment?

The "normal", non-Qiime workflow looks like this:

1. (Uncommon) Convert reference alignment from FASTA to ARB format
2. (First time only) Build Suffix Trie from reference Alignment ("pt server", .arb.pt file).
3. Align sequences to match reference alignment

Step 2 can take quite a while, but since we usually work with a static reference database, the hour or so spent is only expended once.

Questions:

• How do I handle the ARB files? Do I create a new semantic type for those? They are unusual in that they can contain many alignments and trees as well as per-sequence and per-column meta-data (e.g. taxonomy, habitat, etc or variability masks, folding structures, respectively). I'm not sure how those would fit into the semantic type model.
• How do I handle caching of indices? It'd be wasteful to spend 99% of the compute time to re-calculate those. Plus, they are big, so even zipping/unzipping takes enough time to annoy users.

[epruesse]

q2lint wants me to claim you guys wrote the _sina.py. Filed an issue here qiime2/q2lint#20

[epruesse]

ping @gregcaporaso - could you merge #56?

The biggest chunk it adds in terms of disk is libboost. If that's still too much, I could look into doing yet another version of the sina package using static linking. I'd prefer to keep using the lib though.

W.r.t. the ARB data type problem: I've worked around it by accepting either FeatureData[AlignedSequence] as normal --i-reference input parameter or a string --p-arb-reference pointing to an ARB file. Not ideal as the latter breaks providence tracking, but as a trade-off for speed it's workable until a better solution presents itself.