Storage providers

[tdudgeon]

This issue tries to document some of the discussions that we’ve been having about storage in the data tier.

There is a conflicting concern regarding storage of large data files.

1. It’s preferable to have the data local as it can be instantly accessed and operations can be transactional.
2. We want to limit the amount of local storage a user can use because of cost and capacity concerns.

We also want to be flexible in how we handle storage

1. We provide a basic mechanism for storing data but the free tier must be limited in capacity.
2. Pay tiers (bronze, silver, gold ...) can have higher storage quotas and will have a corresponding cost.
3. We want the option of ‘remote storage’ where the data is either left in place, or the data is moved to remote storage such as S3 or Dropbox. We hold the metadata locally but go to the remote store to fetch the file when needed.
4. We would like to allow a user to provide their own remote storage (e.g. S3 or Dropbox) which would bypass quotas for local storage.

To permit this we probably need to introduce the concept of a ‘storage provider’ that allows us to decouple the storage details from the data app.