Comment on Advice - Getting started with LLMs
xcjs@programming.dev 1 month agoOk, so using my “older” 2070 Super, I was able to get a response from a 70B parameter model in 9-12 minutes. (Llama 3 in this case.)
I’m fairly certain that you’re using your CPU or having another issue. Would you like to try and debug your configuration together?
BaroqueInMind@lemmy.one 1 month ago
I think I fucked up my docker setup and will wipe and start over.
xcjs@programming.dev 1 month ago
Good luck! I’m definitely willing to spend a few minutes offering advice/double checking some configuration settings if things go awry again. Let me know how things go. :-)
BaroqueInMind@lemmy.one 1 month ago
My setup is Win 11 Pro ➡️ WSL2 / Debian ➡️ Docker Desktop (for windows)
Should I install the nvidia drivers within Debian even though the host OS already has drivers?
xcjs@programming.dev 1 month ago
I think there was a special process to get Nvidia working in WSL. Let me check… (I’m running natively on Linux, so my experience doing it with WSL is limited.)
docs.nvidia.com/cuda/wsl-user-guide/index.html - I’m sure you’ve followed this already, but according to this. It looks like you don’t want to install the Nvidia drivers, and only want to install the cuda-toolkit metapackage. I’d follow the instructions from that link closely.
You may also run into performance issues within WSL due to the virtual machine overhead.