Are direct resumable uploads to Google Cloud Storage supported?

[gustaff-weldon]

I keep evaluating SFTPGo and testing uploads to Google Cloud storage. I have deployed oss version of sftpgo to kubernetes and gave permissions to GCS bucket.

The folder to upload to is mapped as a virtual folder. Users will have more than one of such virtual folders mapped to different paths in the bucket. Testing with one atm.

The issue I have, is that the uploaded files do not show up in the bucket until upload is finished. Not ideal, if upload gets interrupted on client, or server gets evicted.

Things I have tried that did not work:

• upload_mode=0 (default) with temp_folder, user home folder, virtual folder's mapped_path all set to /transfers-gcs on the folder in the bucket - this results in pipeXYZ file errors and no uploads can be done at all
• upload_mode=0 with temp_folder, user home folder, virtual folder mapped_path all set to /transfers volume mounted to k8s pod - uploads work, but not showing up in bucket until finished. And if client dies, nothing shows up
• upload_mode=8 with temp_folder, user home folder, virtual mapped_path all set to /transfers volume mounted to k8s pod - uploads work, but not showing up in bucket until finished. And if client dies, partial file shows up in bucket, but if server dies, nothing shows up

What I'm looking for is direct upload to bucket via virtual folder, that immediately shows the file as it is being uploaded, so no matter if server or client dies, or encounter an error, partial file is stored and can be resumed.

Can this be achieved? If so, how?

[gustaff-weldon]

cc @drakkan, do we need a version different than OSS or a license for the above to work?

[fostermi]

I believe that sftpgo will always write the file to a user's local home directory /srv/sftpgo/data no matter what the upload configuration is and then move the file to the backend s3 bucket (which seems like a necessity for the aws sdk). See #692.

[drakkan]

I believe that sftpgo will always write the file to a user's local home directory /srv/sftpgo/data no matter what the upload configuration is and then move the file to the backend s3 bucket (which seems like a necessity for the aws sdk). See #692.

Just for clarification: in SFTPGo Enterprise, we’ve implemented memory pipes that allow uploads and downloads to cloud backends without creating temporary files on disk and with minimal memory usage. You can also check the comparison table in the README.