Support for direct access to S3 resources

[sahuguet]

Assuming this is not implemented yet (I could not find it in the doc), I would like to reference s3 resources directly without having to download them locally.

A good example is https://duckdb.org/docs/extensions/httpfs.html where S3 resources can be referenced seamlessly from the query.

[luigi-asprino]

Hi,

You're right. It's not implemented yet, but it would be very nice to have it!
I'm not familiar with S3 resources, but I assume there should be a way to open an input stream from them.
There seems to be a Java library for this https://github.com/aws/aws-sdk-java/tree/master

[luigi-asprino]

great, my personal email is luigi.asprino@gmail.com

[MathiasVDA]

yes I know, you already have mail :)

[luigi-asprino]

I transform this discussion into a issue to keep track of the development.

[MathiasVDA]

Here's a small python snippet that works on my computer after pip install pandas boto :

"""
Simple example for accessing MinIO S3 storage and reading CSV
"""

import boto3
import pandas as pd
import io

# Please fill in your MinIO credentials
MINIO_USERNAME = ""  # Your MinIO access key ID
MINIO_PASSWORD = ""  # Your MinIO secret access key

def simple_s3_csv_reader():
    """Simple function to read CSV from MinIO S3"""
    
    # Create S3 client
    s3 = boto3.client(
        's3',
        endpoint_url='https://minio.matdata.eu',
        aws_access_key_id=MINIO_USERNAME,
        aws_secret_access_key=MINIO_PASSWORD
    )
    
    try:
        # Download the file
        response = s3.get_object(Bucket='sparql-anything', Key='some.csv')
        
        # Read CSV into pandas DataFrame
        df = pd.read_csv(io.BytesIO(response['Body'].read()))
        
        print("CSV loaded successfully!")
        print(f"Shape: {df.shape}")
        print("\nFirst 5 rows:")
        print(df.head())
        
        return df
        
    except Exception as e:
        print(f"Error: {e}")
        return None

if __name__ == "__main__":
    if not MINIO_USERNAME or not MINIO_PASSWORD:
        print("Please fill in your MinIO credentials first!")
    else:
        df = simple_s3_csv_reader()

[enridaga]

I transform this discussion into a issue to keep track of the development.

This feature was added in #560