For about $2000, I picked up a Mac Studio with 96 gigs of RAM, which is effectively all VRAM thanks to Apple’s architecture. It doesn’t have the raw number-crunching power of the big GPUs but with all that space, you don’t need to worry much about the size of the model until they start getting really big, so it’s pretty easy and flexible.
I’m able to do basically everything that others are doing with AI, entirely locally. It generates text and writes code (it’s no Opus, but probably on par with the best of a year and a half ago), images, videos, songs, all that stuff (those last few are garbage but it’s basically the same level garbage that the cloud models are making). And I have total privacy, will never get a surprise price hike or lose access to a model I like, and know exactly how many bottles of water I’m using to cool it (zero).
It’s not heroin, it’s weed. There will be a market for it, but, like, you can also just make it yourself. 90% of what people want will just get done on device. I can’t see any way this turns into a trillion dollar industry.
p03locke@lemmy.dbzer0.com 1 week ago
I use Claude at work and local LLMs at home, and they all produce good code. The Kagi agents work pretty well at searching for information that would have taken me 30-60 minutes to find. Although, I generally favor thinking models because they are good at edge cases.
You have to know how to provide good context for the situation, like examples, prior art, documentation, etc. Many people have a hard time even expressing an idea to a group of humans. Imagine your (pointy-haired) boss shows your department a picture in the next meeting:
B: “Go make this thing!”
D: “What thing?”
B: “Here. This thing. Make something just like it for our company!”
D: “Well, we can’t just copy it outright. What color should it be? What do we want to improve on? How do we tie it to our existing software?”
B: “I don’t know. That’s your job to figure out, right?”
That’s how most people treat LLMs. Garbage in, garbage out.
IchNichtenLichten@lemmy.wtf 1 week ago
Well there you go. Another argument in support of AI being not nearly as big of a deal as the fanboys would have people believe.
If you blame the users (e.g. Steve Job’s “you’re holding it wrong”) it rarely ends well.
p03locke@lemmy.dbzer0.com 1 week ago
Depends on what you mean by “fanboys”. No, the hype isn’t as real as these rich assholes make it out to be. LLMs aren’t going to replace all human workforce everywhere ever, like some of these techbro dipshits quickly find out.
But, I’m still going to use technology that produce a tool in two minutes what would have normally taken me two hours to do. Sure, I have to code review it, but that doesn’t take nearly as long as the work itself.
IchNichtenLichten@lemmy.wtf 1 week ago
I heard someone describe LLMs as useful in the context of being handy plugins which tallies with what you’re saying. In certain situations they’re useful. The problem is that they’re being completely misrepresented and sold as being able to do things that they can’t do consistently or well so the hype and astronomical sums of money being thrown around is problematic.