Skip to content

S3 channel URLs are corrupted due to urljoin not handling s3:// scheme #866

@jezdez

Description

@jezdez

Description

When using S3 channels (e.g., s3://bucket-name/linux-64), package URLs are corrupted to .filename instead of the full S3 URL.

Root Cause

In shards.py, the base_url property uses Python's urllib.parse.urljoin:

@property
def base_url(self) -> str:
    return urljoin(urljoin(self.url, self._base_url), ".")

However, urljoin doesn't recognize s3:// as a valid URL scheme:

>>> from urllib.parse import urljoin
>>> urljoin('https://repo.anaconda.com/pkgs/main/linux-64', '.')
'https://repo.anaconda.com/pkgs/main/'  # Correct!

>>> urljoin('s3://minio-s3-server/linux-64', '.')
'.'  # WRONG - should be 's3://minio-s3-server/'

Then in index.py:621, the URL is constructed via string concatenation:

url=f"{base_url}{filename}"

When base_url is ., this produces .zlib-1.2.11-h7b6447c_3.tar.bz2 instead of s3://bucket/linux-64/zlib-1.2.11-h7b6447c_3.tar.bz2.

Error Message

conda.CondaMultiError: No connection adapters were found for '.zlib-1.2.11-h7b6447c_3.tar.bz2'

Steps to Reproduce

  1. Set up a local S3 server (e.g., Minio)
  2. Create a conda channel on S3
  3. Try to install a package from the S3 channel with libmamba solver
conda create -n test --channel s3://bucket-name zlib

Workaround

Use the classic solver:

CONDA_SOLVER=classic conda create -n test --channel s3://bucket-name zlib

Suggested Fix

The base_url property needs to handle URL schemes that urljoin doesn't recognize (like s3://, ftp://, etc.). One approach:

from urllib.parse import urlparse, urljoin

@property
def base_url(self) -> str:
    parsed = urlparse(self.url)
    # urljoin doesn't handle non-http schemes properly
    if parsed.scheme and parsed.scheme not in ('http', 'https', ''):
        # For non-http schemes, manually construct the base URL
        if self._base_url:
            return urljoin(self.url, self._base_url)
        # Ensure trailing slash for proper concatenation
        return self.url.rstrip('/') + '/'
    return urljoin(urljoin(self.url, self._base_url), ".")

Environment

  • conda version: 26.1.0
  • conda-libmamba-solver version: (current)
  • Python: 3.13

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions