updates to v1.0.5 including sto support

Author: Aarya Venkat

.DS_Store

@@ -71,3 +71,7 @@ site/
 # Claude Code
 CLAUDE.md
 
+# Paper and drafts
+paper/
+REVISION_SUMMARY.md
+

.gitignore

@@ -2,10 +2,90 @@
 
 All notable changes to GapClean will be documented in this file.
 
+## [1.0.5] - 2026-04-05
+
+### Added
+- **Multi-format support** - Stockholm, Clustal, PHYLIP formats now supported
+  - Auto-detect input format from file extension or content
+  - Convert between formats: Stockholm (.sto), Clustal (.aln), PHYLIP (.phy), FASTA (.fa)
+  - New `--input-format` and `--output-format` CLI flags
+  - New `input_format` and `output_format` parameters for `clean_alignment()` API
+  - Automatic format conversion handled internally
+- **Format auto-detection** - No need to specify format in most cases
+  - Detects Stockholm (`# STOCKHOLM` header)
+  - Detects FASTA (`>` header)
+  - Detects Clustal (`CLUSTAL` header)
+  - Detects PHYLIP (numeric first line)
+  - Falls back to extension-based detection
+- **Progress bars for format conversion** 
+  - Shows tqdm progress when converting TO FASTA (per-sequence progress)
+  - Time estimates and warnings when converting FROM FASTA to slow formats
+  - Prevents users from canceling during long Stockholm conversions
+
+### Changed
+- **Added BioPython dependency** (>=1.80) for format conversion
+- **Always defaults to FASTA output** for performance
+  - Stockholm/Clustal/PHYLIP writers are very slow for large alignments (100K+ sequences)
+  - Use `--output-format` explicitly if you need non-FASTA output
+  - Warning displayed when requesting non-FASTA format for large datasets
+- Format validation now happens after conversion to FASTA
+- Temp file cleanup includes converted input files
+- Warning message if `--output-format` conflicts with file extension
+
+### Fixed
+- **Stockholm format parsing** - Pfam alignments now work correctly
+- **Output format auto-detection** - Respects output filename extension instead of defaulting to input format
+- Proper sequence counting for non-FASTA inputs
+- Better error messages for format conversion failures
+
+### Examples
+
+```bash
+# Stockholm input → FASTA output (auto-detected, fast)
+gapclean -i pfam_seed.sto -o cleaned.fa -t 75
+
+# Stockholm input → Stockholm output (explicit, slow for large alignments)
+gapclean -i pfam_seed.sto -o cleaned.sto --output-format stockholm -t 75
+
+# Python API with Stockholm (defaults to FASTA output)
+from gapclean import clean_alignment
+stats = clean_alignment('pfam.sto', 'cleaned.fa', threshold=70)
+```
+
+### Performance Note
+
+**Stockholm/Clustal/PHYLIP output is significantly slower** than FASTA for large alignments due to BioPython's writer implementations. For the Pfam GT2 family (157K sequences):
+
+- FASTA output: 11 seconds total
+- Stockholm output: ~5+ minutes just for format conversion
+
+**Recommendation:** Always use FASTA output unless you specifically need the metadata/markup of other formats.
+
+## [1.0.4] - 2026-04-05
+
+### Added
+- **High-level Python API**: New `clean_alignment()` function for easy programmatic use
+  - Simple one-function interface: `from gapclean import clean_alignment`
+  - Automatic temp file management (no manual cleanup needed)
+  - Returns statistics dict with alignment metrics and timing
+  - Optional `verbose` parameter for quiet operation in pipelines
+  - Full docstring with examples for all modes
+- Exception classes now exported from package root for better error handling
+- Comprehensive example Jupyter notebook using the new API
+
+### Changed
+- Updated visualization tutorial to use new `clean_alignment()` API
+- Cleaner imports: `from gapclean import clean_alignment` instead of internal modules
+
+### Improved
+- Much easier integration into Python scripts and Jupyter notebooks
+- No need to manage temporary files manually
+- Pythonic API design following best practices
+
 ## [1.0.3] - 2026-04-05
 
 ### Added
-- Entropy-based gap removal mode (`-e/--entropy` flag)
+- Entropy-based gap removal modes (`--entropy-min` and `--entropy-max` flags)
 - Comprehensive input validation with clear error messages
 - Type hints throughout codebase
 - Comprehensive test suite (48+ tests)

CHANGELOG.md

@@ -1,4 +1,4 @@
-# GapClean (v1.0.3) 
+# GapClean (v1.0.5) 
 ###### Written by Aarya Venkat, PhD
 
 <img src="gapclean.png" width="350">
@@ -16,22 +16,43 @@ GapClean is a memory-efficient tool for cleaning gappy multiple sequence alignme
 - **Seed Mode**: Remove gaps relative to a reference sequence
 - **Entropy Mode**: Remove low-diversity columns using Shannon entropy
 
-**NEW in v1.0.3:**
+**NEW in v1.0.5:**
+- **Multi-format support**: Stockholm, Clustal, PHYLIP, FASTA
+- Auto-detect input format (no flags needed!)
+- Convert between formats seamlessly
+- **Works with Pfam alignments** (.sto files)
+
+**Also in v1.0.4:**
+- Simple Python API for notebooks and scripts
+- Auto-managed temp files (no manual cleanup!)
+- Returns statistics dict with metrics and timing
+
+**And v1.0.3:**
 - Entropy-based gap removal mode
-- Pure Python implementation (Windows compatible, no `awk` required!)
-- Comprehensive input validation
-- Full type hints
-- Extensive test coverage
-- Professional documentation site
+- Pure Python implementation (Windows compatible!)
+- Comprehensive testing and documentation
 
 ## Features
 
 - **Memory Efficient**: Process alignments larger than RAM using 2D chunking
 - **Fast**: Optimized NumPy operations for gap detection
+- **Scalable**: Handles million-sequence datasets (1M+ sequences in 35 seconds)
 - **Cross-Platform**: Works on Windows, macOS, and Linux
 - **Flexible**: Three gap removal modes for different use cases
 - **Well-Tested**: 48+ tests ensuring reliability
 
+## Performance
+
+Benchmarks on Apple M1 (16 GB RAM) using Pfam protein families (70% gap threshold):
+
+| Scale | Dataset | Sequences | Positions | Time | Size |
+|-------|---------|-----------|-----------|------|------|
+| Tiny | PF08087 (Conotoxin) | 42 | 36 | <0.1s | 2 KB |
+| Medium | PF00535 (GT2) | 157,052 | 1,285 | 11s | 206 MB |
+| **Large** | **PF00069 (Kinase)** | **1,051,876** | **3,667** | **35s** | **7 GB** |
+
+Processing time scales linearly with alignment size (R² = 0.998). From tiny families (42 sequences) to million-sequence datasets with negligible overhead.
+
 ## Installation
 
 ```bash
@@ -55,9 +76,13 @@ gapclean -i input.fa -o output.fa -s 0
 ```
 
 ### Entropy Mode (NEW!)
-Remove low-entropy (conserved) columns:
+Remove columns based on diversity:
 ```bash
-gapclean -i input.fa -o output.fa -e 1.5
+# Keep variable regions (SNP detection)
+gapclean -i input.fa -o output.fa --entropy-min 1.0
+
+# Keep conserved regions (alignment cleaning)
+gapclean -i input.fa -o output.fa --entropy-max 1.5
 ```
 
 ## Usage
@@ -70,9 +95,10 @@ Required Arguments:
   -o, --output   Output cleaned FASTA file
 
 Gap Removal Mode (choose one):
-  -t, --threshold    Percentage threshold (0-100)
-  -s, --seed         Seed sequence index (0-based)
-  -e, --entropy      Entropy threshold (bits)
+  -t, --threshold      Percentage threshold (0-100)
+  -s, --seed           Seed sequence index (0-based)
+  --entropy-min        Remove columns with entropy < threshold (keep variable)
+  --entropy-max        Remove columns with entropy > threshold (keep conserved)
 
 Optional Arguments:
   --row-chunk-size   Sequences per chunk (default: 5000)
@@ -97,16 +123,85 @@ gapclean -i variants.fa -o positions.fa -s 0
 ### SNP Detection
 ```bash
 # Keep only variable positions (DNA)
-gapclean -i population.fa -o snps.fa -e 1.0
+gapclean -i population.fa -o snps.fa --entropy-min 1.0
+```
+
+### Stockholm Format (Pfam)
+```bash
+# Auto-detects Stockholm input, outputs FASTA (fast - recommended!)
+gapclean -i PF00535.sto -o cleaned.fa -t 70
+
+# Stockholm output (slower - only if you need the metadata)
+gapclean -i pfam_seed.sto -o cleaned.sto --output-format stockholm -t 75
+
+# Explicit format specification
+gapclean -i pfam_seed.sto -o cleaned.txt -t 75 --output-format fasta
+```
+
+### Format Conversion
+```bash
+# Convert Clustal to FASTA while cleaning (recommended - fast)
+gapclean -i alignment.aln -o output.fa -t 50
+
+# Convert FASTA to Stockholm (slower for large alignments)
+gapclean -i input.fa -o output.sto --output-format stockholm -t 75
 ```
 
+**Performance Note:** Stockholm/Clustal/PHYLIP output is much slower than FASTA for large alignments (100K+ sequences) due to BioPython's format writers. GapClean defaults to FASTA output for optimal performance. Use `--output-format` only if you specifically need non-FASTA formats.
+
 ### Memory-Constrained Systems
 ```bash
 # Process large alignment with limited RAM
 gapclean -i huge_alignment.fa -o cleaned.fa -t 75 \
   --row-chunk-size 1000 --col-chunk-size 1000
 ```
 
+## Python API
+
+Use GapClean programmatically in Python scripts and Jupyter notebooks:
+
+```python
+from gapclean import clean_alignment
+
+# Threshold mode - remove columns with >50% gaps
+stats = clean_alignment(
+    input_file='input.fa',
+    output_file='output.fa',
+    threshold=50
+)
+
+print(f"Removed {stats['columns_removed']} columns")
+print(f"Took {stats['elapsed_seconds']:.1f} seconds")
+
+# Stockholm format (auto-detected)
+stats = clean_alignment('pfam.sto', 'cleaned.fa', threshold=70)
+
+# Explicit format conversion
+stats = clean_alignment(
+    'input.aln', 'output.sto',
+    threshold=50,
+    input_format='clustal',
+    output_format='stockholm'
+)
+
+# Seed mode - remove gaps relative to first sequence
+stats = clean_alignment('input.fa', 'output.fa', seed_index=0)
+
+# Entropy mode - keep only conserved regions
+stats = clean_alignment('input.fa', 'output.fa', entropy_max=1.5)
+
+# Quiet mode for pipelines
+stats = clean_alignment('input.fa', 'output.fa', threshold=75, verbose=False)
+```
+
+### Why use the Python API?
+
+- **One function call**: No temp file management, no subprocess overhead
+- **Multi-format support**: Stockholm, Clustal, PHYLIP, FASTA - auto-detected!
+- **Returns statistics**: Get metrics about the cleaning operation
+- **Perfect for pipelines**: Integrate into larger bioinformatics workflows
+- **Jupyter-friendly**: See the included visualization tutorial notebook
+
 ## Documentation
 
 **Full documentation available at: [https://arikat.github.io/GapClean/](https://arikat.github.io/GapClean/)**