Either you use your CPU and RAM, either your GPU and VRAM
Comment on What can I actually do with 64 GB or RAM?
zkfcfbzr@lemmy.world 6 days ago
I have 16 GB of RAM and recently tried running local LLM models. Turns out my RAM is a bigger limiting factor than my GPU.
And, yeah, docker’s always taking up 3-4 GB.
Mubelotix@jlai.lu 6 days ago
zkfcfbzr@lemmy.world 6 days ago
Fair, I didn’t realize that. My GPU is a 1060 6 GB so I won’t be running any significant LLMs on it. This PC is pretty old at this point.
Mubelotix@jlai.lu 6 days ago
You can run a very decent LLM with that tbh
fubbernuckin@lemmy.dbzer0.com 6 days ago
You could potentially run some smaller MoE models as they don’t take up too much memory while running. I’d suspect the deepseek r1 8B distill with some quantization would work well.
zkfcfbzr@lemmy.world 6 days ago
I tried out the 8B deepseek and found it pretty underwhelming - the responses were borderline unrelated to the prompts at times. The smallest I had any respectable output with was the 12B model - which I was able to run, at a somewhat usable speed even.
nutsack@lemmy.dbzer0.com 6 days ago
vram would help even more i think