Comment on Are you friends with any AI bots?
quediuspayu@lemmy.dbzer0.com 7 hours agoI’ve been curious about running LLMs locally. Mostly about how long it takes to return a response.
Comment on Are you friends with any AI bots?
quediuspayu@lemmy.dbzer0.com 7 hours agoI’ve been curious about running LLMs locally. Mostly about how long it takes to return a response.
brucethemoose@lemmy.world 7 hours ago
I mean, it depends on your hardware and the model’s size/intelligence.
Worst case for me is many seconds of preprocessing followed by 4-5 words a second.
But you can get almost instant responses + way faster than you can read too.