Can one kcpp instance be used for several conversations - same as llama-server?

[alex-ie]

I wonder if kcpp can be used in a manner of llama-server? I mean to have several conversations / chats that share same instance and same GPU VRAM allocation of model weights but its own KV cache (if I misunderstand how llama-server works - please correct me)? I do not recall such feature of kcpp, but maybe I have missed something.

If no such, how easily could it be added? TIA

P.S. I used to run LLMs on CPU and just did --usemmap to run several instances of kcpp. But now I use GPU to help with prefill / prompt processing and even with -ngl 1 I cannot fit many in VRAM.

[henk717]

Sequentially it can for sure, if you have spare ram smartcache will keep context sessions cached better between them.

In parralel thats more experimental and limited to samplers and things llamacpp can also do, thar would be continuos-batching.

[alex-ie]

I usually consider kcpp as a bundle I download in releases. And it includes UI (Lite). I do not see a way to switch between chats (start new) in Lite - unlike llama-server UI where "New Chart" is clearly visible. And I do not like llama UI partly because for chat itself it leaves less screen space - I like Lite and use it with kcpp engine. Is it possible to switch chats in Lite? If yes, how?

[LostRuins]

Yes, Lite has the save slot option as well as the branch option, you can use that to manage branches or multiple conversations and save files. You can also download sessions as .json saves

Also, Kcpp comes with BOTH lite and the llama.cpp UI bundled, http://localhost:5001 as well as http://localhost:5001/lcpp

[LostRuins]

@alex-ie it already does it automatically as a queue by default.

If you mean getting all instances to reply at the same time, try setting --parallelrequests 4 to allow up to 4 responses at once.

[alex-ie]

--parallelrequests 4

Thanks. How to I switch in Kobold Lite between them? Or do I need different front end for that?

If you mean getting all instances to reply at the same time

I do not need for them to work at the same time, I just need KV cache to be intact and not need to re-Prefill when I switch between chats.

[LostRuins]

I just need KV cache to be intact and not need to re-Prefill when I switch between chats.

oh, in that case, don't use parallerequests, instead try enable --smartcache and it should work.