Replies: 3 comments 7 replies
|
just test it and tell us the Benchmark with your system. |
|
The idea of doing a 2.8TB download to prove I can run it, but more likely than not, to be actually usable and figure everything out isn't just a 1 hour project. I was wondering if there was a more reasonable experiment I could run. But it's ok. The issue isn't using my hardware but the overhead of getting this up and running to just do an experiment. I still wonder about my question of: So is your "Routing history" an attempt to mitigate the high churn that likely occurs with Kimi 3? |
|
the k3 weights are actually only half that size, due to being 4 bit quants, so each parameter takes half a byte. a "reasonable experiment" would involve getting another supported model that's smaller. judging from released numbers, i suspect with your hw you could probably get 5-10 t/s out of GLM 5.2, at least when streaming the model from both SSDs. fast enough to get actual work done in interactive sessions, but probably not fast enough to plug into a bloated agent harness like claude code. |
Thanks for the reply. Perhaps a 4bit version of GLM-5.2 744B might not be too painful to download. Then the question will be is whether colibri is easy to install, configure and get running with this model choice and then run the benchmark. So it is time to read the README.md and see what it take to do this. |
|
Well, this was a waste of time. I though colibri would be using the GPU as much as possible, then the cpu/ram and then fetching from disk. My powerful dual 5090's seem to be nothing but an expensive extra 64 GB's of VRam for colibri. Even if its fancy history prediction of usage patterns and overlapping operations which might be great for a laptop to functionally run colibri it isn't something like vLLM or llama.cpp which are designed to get the most out of the GPU's even if colibri has something neat in it. That neat thing in no way makes it a superior production LLM inference engine. A 400GB download for something which isn't usable with or without colibri. @rofl0r After a few minutes of inferencing at about .25 tok/sec with glm-5.2 perhaps after the cache was warmed and history recorded over repeating a workload over and over again colibri could improve that number. But 5 to 10 tok/sec!? I doubt it. colibri doesn't even bother using the memory on my 2nd gpu. I saw only a very little bit of GPU processor usage on one gpu after hitting return on the prompt but once the tokens started crawling out the GPU usage went to 0 for the cuda cores. I'll keep the 400GB of waste for a couple of days hoping someone will tell me I was stupid and not running it with the right options. |
|
What you say is true. A couple 5090s probably won't help much with running a model that is far too large to fit in them. When I use Colibri with a large model mostly on SSD, I opt to not use my GPU as it doesn't seem to help. If you are looking for something you can run as fast as possible on your system, then you would need a model that fits entirely into your system RAM. Then you can see if using the GPUs helps or not. At present Colibri does not support any models that fit into 256GB, but hopefully support will soon be added for Deepseek V4 Flash 0731. In fact, I can already use that model on my computer with 256GB of RAM, using a llama.cpp based application, as it fits entirely into the RAM. When Colibri adds support for that model, we will be able to find out if Colibri is faster than llama.cpp or not. But please understand, that Colibri was originally designed for streaming extremely large models mostly from SSD, which is probably always going to be relatively slow. |
|
You were also not running it with the right options — and the reason you were not is our fault. I went through the code with your machine in mind rather than answering from memory, and there are two separate things, one of which I am certain about and one of which I need your logs to settle. Certain: your GPUs were never given anything to do
is exactly right, and it is not a scheduling subtlety or the history predictor giving up. We put zero experts on your 64 GB of VRAM because that is what we ship as the default. Certain: the resident hot store is off until it has learned your workload
There is an autopin that fills that gap, but it needs history: it does nothing below 5,000 recorded expert selections and scales its quota by Uncertain, and this is what I want your logs forThe adaptive LRU expert cache is sized automatically from Three lines from a run would settle it: the What to runOMP_NUM_THREADS=64 \
URING=1 \
PIN=auto \
COLI_MODEL_MIRROR=/mnt/second-t705/glm52 \
coli chat --auto-tier --gpu auto --vram 60 --model <glm52-int4-g64>
Then run it again. The usage file accumulates across sessions, so the second and third conversations are meaningfully faster than the first. A single cold run on this engine measures the disk, not the engine. Check the banner first. It prints an The estimate, with the arithmetic rather than the conclusionThe last number I gave in this thread was 4–6 tok/s and you measured 0.25, so here is the reasoning to check rather than a figure to trust. Fast memory: 64 GB VRAM + ~200 GB usable RAM ≈ 264 GB. Dense is ~10 GB, leaving ~254 GB against ~362 GB of experts — roughly 70% residency, warm. Decode moves on the order of 11 GB of expert weight per token at zero hit; at 70% that is ~3 GB/token still coming off NVMe, which two striped Gen5 drives should serve in something like 0.3 s. The resident 70% comes from 8-channel DDR5, fast enough not to be the limit. 2–4 tok/s warm is where that lands. Calibration, all measured and public:
You sit under those because you are at ~70%, not 100%. Call it ten times where you are now, with the honest caveat that ten times 0.25 is still 2.5. Which is not a coding agent, and I am not going to pretend it is. 10,000 tokens at 3 tok/s is nearly an hour. @rofl0r's 5–10 was optimistic. Your 0.25 was a misconfiguration. The truth sits in between and it does not close an agent loop. Where your box genuinely wins: batch passes over a repository, evaluations that have to be byte-reproducible in two years, and instrumenting a 744B model in ways no API exposes. The thing you found without looking for it@brad-evony's line is the sharpest criticism in this thread and it is correct:
OLMoE 4 GB, GLM-5.2 372 GB, Inkling 469 GB, Kimi K3 1.6 TB. Nothing between 4 and 372. The 256 GB workstation — the machine most likely to want this — has nothing that fits, gets pushed into the streaming regime, which is our worst case, and reasonably concludes the engine is slow. DeepSeek V4 Flash at int4 lands near 140 GB and would sit entirely inside your RAM with the GPUs on top. That, not a flag, is what would make your hardware sing, and it is in review now. What I am askingKeep the 400 GB for a few more days if you can. Post those three lines and I will read them properly — and if the warm number lands materially under 2, that is a bug report I want more than a good benchmark, because it means the automatic cache sizing is not doing what I just told you it does. And the measurement I would most like from you specifically: whether two matched Gen5 drives actually sum under striping. That sits in our open-hypotheses table with "the bandwidth model is sound" next to it and no datapoint behind it. Nobody who has reported here had the hardware to check. |
|
Let me know about the test with new command. |
@JustVugg Thanks for that detailed reply. Just a heads up. Just blindly trying the better args to the command you suggested got me to 3.03 tok/sec and further tuning should increase that. Now this is getting interesting. I need to do a clean benchmark to get you good numbers. Kill my chrome running about 100 tabs, purge the OS buffer cache, see how much free ram I have to set --ram to the largest value which doesn't cause my kswapd0 to start showing up with 'top', bind the 64 workers threads to my 64 cores(might help a bit). Do you actually have a consistent benchmark that you used for your published numbers. Something like, from a clean slate, 5 consecutively issued prompt to measure the increase tok/sec until it stabilizes? As for as avx2 vs avx512-vnni goes I don't see an idot: line nor either of these in the startup message. Perhaps stupid but the actual flag name on my threadripper is avx512_vnni. I need to check your code to see if it is testing and enabling it properly. FYI, I was investigating uring, io queue depth, IO speed of random reads vs sequential reads, directio, fadvise and so many other things in commercial sql databases for a long time before retiring. Now I do AI as a hobby. I hope to give a further status later today. I'm in PST timezone. |
|
Couldn't make it faster with anything I tried. Running any test is easy but I'm not sure "exactly" the test you want done nor what you mean by those "3 lines" or where the log output is that your asking for. It would require some system instrumentation and analysis to figure out if things could be stretched to 4 or 5 tok/sec from what I've getting. Because I'm sure what your prompts were do a proper benchmark I had fun with the five prompts below which I ran one by one: What would Plato say to all of this? What would Bertrand Russell say to all of this? Summarize these 3 view points. Would a donkey have any opinion on all of this? glm-5.2 is good even if slow on my box. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I primarily research training of tiny/small models. The 124MB modded-nanogpt model trains in under 15 minutes from scratch but while amusing and coherent it isn't useful. So I'm looking into models from 200MB to 4GB. But that is small model training.
When I get bored, with the above, would it be relatively easy to see how big of a model my system could run inference on?
While it appears that in theory I could run the 2.8T model, at perhaps something like ?1 tok/sec, I'd never use it to generate a 10000 token application and have it iterate on fixing bugs until it got it right.
What might be the sweet spot of a large good model a machine like mine could run with a performance that allows it to be useful as a coding agent? Yeah, there are some great 32B and 80B model classes but I keep thinking I can run something even bigger.
NOTE: I'm a retired performance architect. When I looked at MoE, on paper before, I wondered if the concept of a hot working set comes into play. It appeared that was the case. Then I researched the huge new 2.8 Kimi yesterday and it appears that it is a different kind of ?fine-grained? MoE model and that for any small to medium number of tokens it could very well need to touch nearly all the 896 experts/shards(?) of the Kimi 3 model. Is that true? I just found colibri minutes ago... So is your "Routing history" an attempt to mitigate the high churn that likely occurs with Kimi 3?
All reactions