This is my upgraded rig:
- Ryzen9 5950x with 64gb DDR4
- Dual NVIDIA RTX A4000 (16+16GB VRAM)
to whom read my previous posts, i jumped the gun and upgraded my server, it was worthwhile and somewhat cheap given i already had the two GPUs and the DDR4 RAM.
Anyway, i am currently running Qwen3.6-35B-A3B-UD-Q5_K_XL all in VRAM with 65536 context and pretty happy with speed (80-90t/s) and overall responses (mostly chat).
I would like to experiment with something beefier, with CPU offload, that i can run with my llama.cpp. Of course t/s is not a goal here, but precision and accuracy of responses is.
I tried to find a good model with claude and gemini, but always got short. Once the model suggested fully crashed my server (guess fill up RAM and ended up in a swap loop), more then once i ended up chasing non existent models. Pretty annoying.
Considering i would only use between 32 and 48GB or system RAM, can you suggest (preferably with links to HF) some models?
I like qwen3.6, but open to anything.
For sure. Feel free to ask questions, too.
The TL;DR I will leave you with is this; some of what we consider as “smarts” in a LLM has traditionally done by brute force - bigger GPU , more parameters.
The alternative approach is to make the llm do less by itself, but instead, call on other tools. That way, you can squeeze out much more from a smaller llm or weaker hardware, so long as the llm is obedient at tool calling.
Think of it like doing arithmetic in your head vs using a calculator. Both provide the answer, but the latter requires much less brain power.