Automatic zip/unzip file content on saving and restoring using activerecord plugin

[pldavid2]

I’m using shrine with activerecord plugin in a pure ruby application.

Is there a recommended way so content can be automatically zipped when stored, and unzipped when retrieving it?
Checking the docs seems I can manually zip the content using the process(:store) directive (Processing | Shrine) but not sure if there is something similar on the retrieve for unzipping.

Maybe there is already some feature/plugin that do this?

Thanks in advance

[pldavid2]

I’m still trying to implement this but not sure how. Anyone knows a way to do it?

[pldavid2]

Hello and happy new year. I still need something like this, zip on upload/store and unzip on retrieval.
Maybe @janko you have some suggestion?

[janko]

Hi, happy new year 🙂 I don't know of anything, processing original files was deprecated in Shrine 3 (i.e. the processing plugin), and there isn't really an official replacement.

You could try overriding Shrine#upload and Shrine::UploadedFile#open methods. Or better yet, you could create your own storage wrapper, where Storage#upload zips and Storage#open unzips. Yeah, that's probably a better abstraction layer to plug this in.

class CompressedStorage < SimpleDelegator
  def upload(io, ...)
    io = zip(io)
    super
  end

  def open(...)
    io = super
    unzip(io)
  end

  private

  def zip(io)
    # zipping logic
  end

  def unzip(io)
    # unzipping logic
  end
end

Shrine.storages = {
  cache: CompressedStorage.new(Shrine::Storage::S3.new(...)),
  store: CompressedStorage.new(Shrine::Storage::S3.new(...)),
}

[jrochkind]

I think for performance and you'd want to zip/unzip in a streaming fashion (also for full compat with shrines streaming api), which would require a bit more code. I think the ruby zip library supports streaming zip/unzip, but I'm not totally sure. if it doesn't, it would be a barrier to this, but not on shrine's end.

if it does, ruby's lack of a totally documented standard stream API can make stream interop somewhat more confusing than it should be, if the Down object shrine uses doesn't provide the full API zip expects in a stream, you may have to provide some glue. And it will also take a bit more insight on the shrine end just taping it all together.

If you can't do a stream-supporting integration, then if you have any shrine clients trying to stream, they are going to get a delay while the entire download is buffered and then unzipped. Which might still be worth it to you? It could still work with that caveat.

It would be an interesting project to try! i believe shrine will provide the tools to do it, if it can be done with zip. Definitely agree a custom 'storage' is the way to go for what you spec'd.

[pldavid2]

Thank you both!
I like the idea of the custom storage wrapper. I'm trying to save this zipped content in a compatible S3 storage, but not AWS so can't use lambdas for the zip/unzip tasks.
Not sure if streaming can be done with rubyzip but found zip_kit/zipline alternative gems that seems to provide that. Not a fan of adding extra gems but maybe to start working could help.

[pldavid2]

Ended up with something like this, only compressing on the store phase. Not very streaming fashion, because my small knowledge with good streaming practice..

class CompressedStorage < SimpleDelegator
    
  def upload(io, ...)
    io = zip(io)
    super
  end

  def open(...)
    io = super
    unzip(io)
  end

  private

  def zip(io)
    # zipping logic
    zip_stream = Zip::OutputStream.write_buffer do |zip|
      entry = io.metadata['filename']   if io.respond_to?(:metadata)
      zip.put_next_entry(entry || 'entry')
      zip.write(io.read)
    end
    zip_stream.rewind

    zip_stream
  end

  def unzip(io)
    # unzipping logic
    unzipped_io = nil
    Zip::InputStream.open StringIO.new(io.read) do |is|
      _entry = is.get_next_entry
      unzipped_io = StringIO.new(is.read)
    end
    unzipped_io
  end
end

Metadata stored is the original not "zipped", which is fine as I wanted transparent zip/unzip to save space, and keep reference of original size and type of files.

Unfortunately I had to add in the uploader class some glue to set the correct zip content type and extension on S3, as the S3 storage was picking the original content type on uploading from the metadata.

  def generate_location(io, **options)
    location = super
    location = "#{File.basename(location, '.*')}.zip"  if storage_key == :store
    location
  end
 
  plugin :upload_options, store: -> (io, options) do
    { content_type: 'application/zip' }
  end

Not the best implementation for sure, but is currently working.

Thanks for your help and any extra tip will be welcome for sure 🙂