DeepSeek V4 Flash on an RTX 5070 Ti Laptop (12 GB VRAM): CUDA + SSD Streaming Field Results and Implications for Colibri #963
Replies: 4 comments 4 replies
|
I added the gpu support also, normally i get 1.14 tok/s, depends on apps concurrently running, this time got a little lower: PS C:\colibri> $env:COLI_MODEL_MIRROR = "D:\llm_models\DeepSeek-V4-Flash-0731-Abliterated" ▄▄▄▄▀▀▀▀▄▀▀ tiny engine, immense model ╭────────────────────────────────────────────────────────────────────────────────────────────────╮ ◆ colibri It's often used for testing fonts, keyboards, and other text-rendering tools. Is there something specific you'd like to do with this sentence? For example, I could: • Count the letters. Let me know! |
|
squeezed 1.26 tok/sec now, with some code optimizations |
|
Try the new 1.6.0 |
|
what you mean is now GPU integrtated on DS4? didn't saw it in 1.6.0 |
|
no not yet, I read this wrong, it will arrive anyway |
|
i have it working, if you want i can push the branch |
|
Yes if is possible make the PR i merge instantly👍🏻 |
|
i'm currently in testing phase before PR, i will when i'm sure almost all work as it should |
Uh oh!
There was an error while loading. Please reload this page.
I wanted to share a real-world low-VRAM test that may be relevant to Colibri's memory-tiering and SSD-streaming goals.
I tested DeepSeek V4 Flash on a consumer laptop with a 12 GB Blackwell GPU using CUDA + SSD streaming. The model was not fast enough for interactive chat, but it did run successfully, and the cache-budget behavior was interesting.
Environment
sm_120For the test I used the current
antirez/ds4CUDA backend with DeepSeek V4 Flash and SSD streaming.Build:
Initial result
A minimal smoke test succeeded:
./ds4 \ -m ds4flash.gguf \ --ssd-streaming \ --ssd-streaming-cache-experts 12GB \ --ctx 4096 \ --tokens 1 \ --nothink \ -p "Reply with exactly: OK"The runtime automatically capped the requested 12 GiB expert budget to 10 GiB.
The model returned:
The runtime planned approximately:
So the model was technically running with only a small resident portion on the GPU while routed experts were streamed from SSD.
MMQ prefill issue
With a normal prompt, the default MMQ prefill path exceeded the available VRAM:
Disabling that prefill optimization:
made the SSD-streaming configuration stable.
10 GB vs 8 GB expert budget
I tested the same prompt with two cache budgets.
10 GB
DS4_CUDA_MMQ=0 ./ds4 \ -m ds4flash.gguf \ --ssd-streaming \ --ssd-streaming-cache-experts 10GB \ --ctx 4096 \ --tokens 100 \ --nothink \ -p "Explain in Turkish, in a few sentences, what a large language model is."Result:
Planned GPU memory:
8 GB
DS4_CUDA_MMQ=0 ./ds4 \ -m ds4flash.gguf \ --ssd-streaming \ --ssd-streaming-cache-experts 8GB \ --ctx 4096 \ --tokens 100 \ --nothink \ -p "Explain in Turkish, in a few sentences, what a large language model is."Result:
Planned GPU memory:
Comparison:
Unexpectedly, the smaller 8 GB budget was slightly faster while also leaving substantially more VRAM headroom.
This suggests that on a very constrained GPU, maximizing cache residency is not necessarily the fastest configuration. The additional memory pressure near the VRAM limit may outweigh the benefit of keeping more experts resident.
A non-fatal warning remained:
but inference completed successfully.
OpenAI-compatible server test
I also ran the model through an OpenAI-compatible server:
The API worked correctly and I connected it to Open WebUI.
The model produced valid responses, including reasoning output.
However, on this hardware the end-to-end latency is still too high for fluid interactive chat. The setup is more useful as a proof that a very large MoE model can be made functional on constrained consumer hardware than as a practical daily chat configuration.
Why I thought this might be relevant to Colibri
What interested me most was not simply that the model could run, but how strongly the result depended on the balance between:
On this machine, a smaller GPU cache actually produced better measured throughput.
That seems relevant to Colibri's goal of treating VRAM, RAM and storage as different tiers rather than trying to maximize residency at all costs.
I would be very interested to know whether Colibri's planner is moving toward automatically searching for this kind of "sweet spot" on low-VRAM systems instead of simply maximizing the hot tier.
Hope these results are useful as a low-VRAM consumer-hardware datapoint for Colibri's CUDA and SSD-streaming work.
Test:
All reactions