Make retry behavior configurable

[gawbul]

Priority: Low (Enhancement)

Issue

Retry behavior is currently hardcoded with max_attempts = 5 and a fixed backoff strategy. Different use cases might need different retry configurations.

Current Code

# pyensemblrest/ensemblrest.py
self.max_attempts: int = 5

# In __retry_request:
to_sleep = (self.wall_time + 1) * self.last_attempt

Recommendation

Make retry behavior configurable:

class EnsemblRest:
    def __init__(
        self, 
        api_table: dict[str, Any] = ensembl_api_table,
        max_attempts: int = 5,
        backoff_factor: float = 1.0,
        backoff_strategy: Literal["linear", "exponential"] = "linear",
        **kwargs: dict[str, Any]
    ) -> None:
        """
        Initialize EnsemblRest client.
        
        Args:
            max_attempts: Maximum number of retry attempts (default: 5)
            backoff_factor: Multiplier for backoff time (default: 1.0)
            backoff_strategy: Retry backoff strategy - "linear" or "exponential"
        """
        self.max_attempts = max_attempts
        self.backoff_factor = backoff_factor
        self.backoff_strategy = backoff_strategy
        # ... rest of init
        
    def __retry_request(self) -> Any:
        """Retry last request with configurable backoff."""
        self.last_attempt += 1
        
        if self.last_attempt > self.max_attempts:
            raise EnsemblRestError(...)
        
        # Calculate sleep time based on strategy
        if self.backoff_strategy == "exponential":
            to_sleep = (self.wall_time + 1) * (2 ** self.last_attempt) * self.backoff_factor
        else:  # linear
            to_sleep = (self.wall_time + 1) * self.last_attempt * self.backoff_factor
        
        logger.debug(f"Sleeping {to_sleep}s before retry {self.last_attempt}/{self.max_attempts}")
        time.sleep(to_sleep)
        # ... rest of retry logic

Usage Examples

# Conservative: More retries with exponential backoff
ensRest = EnsemblRest(
    max_attempts=10,
    backoff_strategy="exponential",
    backoff_factor=0.5
)

# Aggressive: Fewer retries with shorter waits
ensRest = EnsemblRest(
    max_attempts=3,
    backoff_factor=0.5
)

# No retries (fail fast)
ensRest = EnsemblRest(max_attempts=1)

Benefits

• Flexibility for different use cases
• Better control over API consumption
• Easier to test with reduced retry attempts
• Can optimize for batch vs interactive usage

Files to Update

• pyensemblrest/ensemblrest.py - Add configurable retry
• README.md - Document retry configuration
• tests/ - Add tests for different retry strategies

Testing

• Test with different max_attempts values
• Test linear vs exponential backoff
• Verify backoff_factor works correctly
• Test with max_attempts=1 (no retry)