First you need to get a program that reads and runs the models. If you are an absolute newbie who doesn’t understand anything technical your best bet is llamafiles. They are extremely simple to run just download and follow the quickstart guide to start it like a application
They recommend llava model you can choose from several prepackaged ones. I like mistral models.
Then once you get into it and start wanting to run things more optimized and offloaded on a GPU you can spend a day trying to setup kobold.cpp.
My primary desktop has a typical gaming GPU from 4 yrs ago
Primary fuck around box is an old Dell w onboard GPU running proxmox
NAS has like no gpu
Also have a mini PC running HAOS
And a couple of unused pi’s
Can I do anything with any of that?
Your primary gaming desktop gpu will be best bt for running llms first check your card for exact information more vram the better. Nvidia is preferred but AMD cards work.
First you can play with llamafiles to just get started no fuss no muss download them and follow the quickstart to run as app.
Once you get it running and want better performance or latest models then you can spend some time installing and running kobold.cpp with cublas or vulcan and GPU offloading.
Smokeydope@lemmy.world 3 months ago
First you need to get a program that reads and runs the models. If you are an absolute newbie who doesn’t understand anything technical your best bet is llamafiles. They are extremely simple to run just download and follow the quickstart guide to start it like a application They recommend llava model you can choose from several prepackaged ones. I like mistral models.
Then once you get into it and start wanting to run things more optimized and offloaded on a GPU you can spend a day trying to setup kobold.cpp.
blazeknave@lemmy.world 3 months ago
My primary desktop has a typical gaming GPU from 4 yrs ago Primary fuck around box is an old Dell w onboard GPU running proxmox NAS has like no gpu Also have a mini PC running HAOS And a couple of unused pi’s Can I do anything with any of that?
Smokeydope@lemmy.world 3 months ago
Your primary gaming desktop gpu will be best bt for running llms first check your card for exact information more vram the better. Nvidia is preferred but AMD cards work.
First you can play with llamafiles to just get started no fuss no muss download them and follow the quickstart to run as app.
Once you get it running and want better performance or latest models then you can spend some time installing and running kobold.cpp with cublas or vulcan and GPU offloading.