Why does list iterator generate a Sequence instead of flat ObjectMeta's?

[tooruu]

When I call store.list('prefix') I expect the output to be ListStream[ObjectMeta] but instead it returns a ListStream[Sequence[ObjectMeta]] which contains a singular list of ObjectMeta. Why is it implemented like this?
To access the path of each object I have to do

for seq in store.list('prefix'):  # Only does 1 iteration
    for obj in seq:
        obj["path"]

instead of

for obj in store.list('prefix'):
    obj["path"]

Meanwhile store.list('prefix').collect() returns a flat list of ObjectMeta. What gives?

[kylebarron]

It's like this to express chunking and give insight into the batching and network requests that are happening under the hood.

When you have an AsyncGenerator that yields bare ObjectMeta, you never know when the next async for iteration will be already buffered or cause another network request. But when the AsyncGenerator yields Sequence[ObjectMeta], you are able to inspect the full buffering yourself. If you need more items than exist in the current sequence, then you can call anext(), but you know you're incurring a network request there.

Conversely, collect says "I want to materialize the entire list of results". So at that point you don't care how many requests it took to create it

Does this help?