A question about KV cache

[Togaroda]

Will it be possible to extract/save and retrieve KV cache for different chat sessions?

What I mean is, for example, if you created a regular chat application like Llamatik, and the user created multiple chats at different times, when they open a chat, its KV cache could then be retrieved and saved after the response is generated, right?

[ferranpons]

Hi @Togaroda

Yes, that should be possible.

The intended approach is to keep one KV cache/session file per chat conversation. When the user opens a specific chat, its KV cache can be loaded, generation can continue from that restored state, and then the updated KV cache can be saved again after the response is generated.

So for a regular multi-chat app like Llamatik, each chat could maintain its own persisted KV state.

The main limitations are that the cache must match the same model/configuration, and if older messages in the conversation are edited or removed, the KV cache would need to be invalidated and rebuilt.