feat(cubestore): Add CUBESTORE_S3_ENDPOINT for GovCloud and custom S3 endpoints

[jdaripineni]

Check List

• Tests have been run in packages where changes have been made if available
• Linter has been run for changed code
• Tests for the changes have been added if not covered yet
• Docs have been added / updated if required

Summary

This PR adds support for custom S3 endpoints via a new CUBESTORE_S3_ENDPOINT environment variable, enabling CubeStore to work with AWS GovCloud regions and other S3-compatible services that require explicit endpoint configuration.

Problem

The rust-s3 library used by CubeStore has hardcoded support for standard AWS commercial regions:

pub enum Region {
    UsEast1,      // → s3.us-east-1.amazonaws.com
    UsEast2,      // → s3.us-east-2.amazonaws.com
    UsWest1,      // → s3.us-west-1.amazonaws.com
    UsWest2,      // → s3.us-west-2.amazonaws.com
    // ... more commercial regions ...
    Custom { region: String, endpoint: String },
}

When you set CUBESTORE_S3_REGION=us-gov-west-1, the library's region.parse::<Region>() doesn't recognize GovCloud regions. It falls back to creating:

Region::Custom {
    region: "us-gov-west-1",
    endpoint: "us-gov-west-1"  // ❌ Invalid! This is NOT a URL
}

This results in malformed S3 URLs:

https://bucket.us-gov-west-1/path/to/object  ← BROKEN

Instead of the correct:

https://bucket.s3.us-gov-west-1.amazonaws.com/path/to/object  ← CORRECT

Why not just pass the full URL as the region?

AWS Signature Version 4 uses the region name in the signature calculation:

SignatureV4 = HMAC-SHA256(date, region, service, request)
                                ^^^^^^
                        Must be "us-gov-west-1"

If you pass s3.us-gov-west-1.amazonaws.com as the region, the endpoint URL would be correct, but the signature would be invalid because AWS expects the signature to be calculated with us-gov-west-1.

Solution

Add a new optional CUBESTORE_S3_ENDPOINT environment variable that allows users to specify the S3 endpoint URL separately from the region.

When CUBESTORE_S3_ENDPOINT is set, CubeStore creates:

Region::Custom {
    region: "us-gov-west-1",                           // ✅ For AWS signature
    endpoint: "https://s3.us-gov-west-1.amazonaws.com" // ✅ For S3 URL
}

Both values are correct for their respective purposes:

• region: Used in AWS Signature Version 4 signing
• endpoint: The actual HTTPS URL where S3 API requests are sent

Usage

AWS GovCloud

CUBESTORE_S3_BUCKET=my-bucket
CUBESTORE_S3_REGION=us-gov-west-1
CUBESTORE_S3_ENDPOINT=https://s3.us-gov-west-1.amazonaws.com

AWS China

CUBESTORE_S3_BUCKET=my-bucket
CUBESTORE_S3_REGION=cn-north-1
CUBESTORE_S3_ENDPOINT=https://s3.cn-north-1.amazonaws.com.cn

Custom S3-compatible storage (MinIO, LocalStack, etc.)

CUBESTORE_S3_BUCKET=my-bucket
CUBESTORE_S3_REGION=us-east-1
CUBESTORE_S3_ENDPOINT=https://minio.example.com

Affected Regions

This fix enables support for any region not hardcoded in rust-s3, including:

Region	Description
us-gov-west-1	AWS GovCloud (US-West)
us-gov-east-1	AWS GovCloud (US-East)
cn-north-1	AWS China (Beijing)
cn-northwest-1	AWS China (Ningxia)
Any new AWS regions	Future-proofing
Custom endpoints	MinIO, LocalStack, Ceph, etc.

Changes

rust/cubestore/cubestore/src/config/mod.rs

• Added endpoint: Option<String> field to FileStoreProvider::S3 enum
• Added parsing of CUBESTORE_S3_ENDPOINT environment variable
• Pass endpoint to S3RemoteFs::new()

rust/cubestore/cubestore/src/remotefs/s3.rs

• Added endpoint: Option<String> parameter to S3RemoteFs::new()
• When endpoint is provided, create Region::Custom with proper endpoint URL
• When endpoint is not provided, use existing region.parse() behavior (backward compatible)

rust/cubestore/cubestore/src/remotefs/mod.rs

• Updated tests to include endpoint parameter

rust/cubestore/cubestore/src/sql/mod.rs

• Updated tests to include endpoint: None in FileStoreProvider::S3 structs

Backward Compatibility

This change is fully backward compatible:

• CUBESTORE_S3_ENDPOINT is optional
• When not set, existing behavior is preserved
• All existing deployments continue to work without changes

Testing

• Updated existing S3 tests to include the new endpoint parameter
• Manual testing with AWS GovCloud us-gov-west-1 region confirmed working (can see cubestore objects created in S3 indicating the S3 connection works with the fix)

Test Results

Test Command

cd rust && docker run --rm \
  -v "$(pwd)":/app \
  -w /app/cubestore \
  -e TMPDIR=/app/cubestore/target/tmp \
  rust:latest bash -c "
    mkdir -p /app/cubestore/target/tmp &&
    apt-get update &&
    apt-get install -y cmake libssl-dev pkg-config libsasl2-dev clang libclang-dev lld &&
    rustup default nightly-2025-08-01 &&
    cargo test -p cubestore -- --test-threads=1
  "

Test Summary

test result: ok. 174 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 130.57s

S3-Related Tests

test remotefs::tests::aws_s3 ... ok
test remotefs::tests::gcp_remote_fs ... ok
test remotefs::tests::local_dir ... ok
test remotefs::tests::minio_remote_fs ... ok
test remotefs::queue::test::queue_download_remotefs_error ... ok
test remotefs::queue::test::queue_download_wrong_file_size ... ok
test remotefs::queue::test::queue_upload ... ok
test remotefs::queue::test::queue_upload_missing_file ... ok
test remotefs::queue::test::queue_upload_wrong_size ... ok

Lint Results

Format Check

cargo fmt -p cubestore --check

Result: ✅ PASSED

Clippy

cargo clippy -p cubestore --no-deps -- -D warnings

Note: The clippy errors are pre-existing issues in the cubestore codebase (981 warnings), not introduced by this PR. No new clippy warnings were added by our changes.

Acknowledgments

Claude Opus 4.5 was used to analyze and fix this issue.