Motivation
Currently, Flame implements cache in a naive way: it's only using HashMap via rest api. It does not work for large data, and it's not efficient.
Function Specification
Build a dedicated object cache via Arrow Flight & Polars.
Solutions
- For build-in type, just pickle with
RunnerRequest directly
- For normal object, put to cache after pickle
- For DataSet/DataFrame, put to cache directly
Additional context
N/A
Motivation
Currently, Flame implements cache in a naive way: it's only using
HashMapvia rest api. It does not work for large data, and it's not efficient.Function Specification
Build a dedicated object cache via Arrow Flight & Polars.
Solutions
RunnerRequestdirectlyAdditional context
N/A