Description
delta logs can become fairly large so fully materialising them should be done only when really required.
Whenever we use concat_batches this is likely the case. an example being the add_actions_table method, which currently returns a single record batch.
We should change that method to return BoxStream<...> instead and use the same lazy readers we use for scans to expose it in python.
Description
delta logs can become fairly large so fully materialising them should be done only when really required.
Whenever we use
concat_batchesthis is likely the case. an example being theadd_actions_tablemethod, which currently returns a single record batch.We should change that method to return
BoxStream<...>instead and use the same lazy readers we use for scans to expose it in python.