GitHub - JustVugg/colibri: Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
github.com
Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦 - JustVugg/colibri

I recently come across this project, which seems very interesting because it let you run literally huge models on consumer hardware.

I am fancying yo try run Qwen3.8 flash next with this on my setup. I don’t expect anything usable, but just for the fun of it. What stopping me is the download size of the model.

Has anybody tried colibri? Just to have an idea of speed to expect. 1t/s? 0.1t/s?

  • Sims@lemmy.mlEnglish
    2·
    2 days ago

    Not me. Just searched for tests, and found this guy:

    Quick recap: He tests glm5.2(744gb version) on a mac m2, and a rtx5090 workstation. The very short context tests are not completely the same, he does no warm tests, and he doesn’t test the cluster feature, or any multi ssd setups etc.

    M2 32gb external drive: cold start: ~2minutes ttft (time to first token), 0.1 tps. PC 64gb internal drive + rtx5090: cold start: ~17 seconds ttft, 0.8 tps

    His conclusion is that the amount of memory is still the most important thing as warm runs should cache all experts etc.

    This guy compare simple colibri with llama.cpp with a hidden flag ‘-nr’, and as soon as the context gets longer than ‘What is the Capitol of yadayada’, llama.cpp are much faster, and rising with context length. At 3K context, llama.cpp is 8+ * faster. Std. Colibri stays at same low tps no matter context length, but have a quicker ttft. He has a few links to similar projects in description.

    end of test-vid recap…

  • hendrik@palaver.p3x.deEnglish
    2·
    3 days ago

    They have some numbers in their Readme file. Seems to me for the model they tested, it’s number of beefy graphics cards = Tokens/s…
    Their rig with 6x RTX 5090 does about 6T/s, their laptop with one RTX5070 does 1T/s. And their computer without a GPU does about 0 T/s.