Giant cli: spin off upload operations to aws s3 sync?

[hoyla]

Sequential upload to the ingestion-in s3 bucket via the scala tool is very slow.

aws s3 sync is much faster: could we sync to a staging s3 bucket, and have something copy those files into the in bucket, so sync can continue at speed without stuff disappearing?

Workflow could be:

1. Kick off an aws sync to the staging s3 bucket - no interaction with Giant required at all
2. Use cli to create an ingestion and tell a service to start moving files from the staging bucket folder into the ingestion inbox
3. Once aws s3 sync is complete the user can leave the cloud services to their own devices
4. Possibly at some point later clean up old material from the staging bucket.

We could optionally tell the service at step 2 to use any aws s3 bucket (or Google Drive etc), given the right credentials. This would allow us to ingest stuff without ever downloading locally to a user device.

Useful features in various flavours of the cli:

Present in scala cli on main:

• Authenticate
• Create collection/ingestion
• Bulk create users
• Make raw authenticated api calls
• Upload to an ingestion from wherever you run the command from
• Compute the pfi hash of a file
• Debug modes: verbose; dry-run etc
• List all collections and their ingestions
• Delete an ingestion

Present in scala client on ljh-client-dev

• Upload with checkpointing to faciliate resumption
• Generate new checkpoint file from Giant index
• Details about a collection - its ingestions, top level folders, file counts, etc
• Verify that the contents of a local path match the indexed files for a collection, ingest or ingest node (folder, file)
• Delete a collection and all its contents
• Delete blobs within an ingestion (e.g. at a subpath)

Not present:

• Download a collection, ingest, or ingest node (folder, file)**
• Upload to an ingestion from elsewhere (S3, Google Drive etc)**