Comment

Unforeseen@sh.itjust.works ⁨1⁩ ⁨day⁩ ago

The efficiency improvements in some open models are becoming crazy, like hundreds of times from a year ago. I have a setup such as yours on my framework which can handle a 120b param model fully loaded. It’s capable of the RAG setup you are already envisioning.

source

Sort:hotnew top

Illecors@lemmy.cafe ⁨1⁩ ⁨day⁩ ago
Moar deets, please!

source
cornshark@lemmy.world ⁨1⁩ ⁨day⁩ ago
What do you mean framework? What are the specs?

source
- Unforeseen@sh.itjust.works ⁨1⁩ ⁨day⁩ ago
  Hi,
  
  I’m running the AMD AI 395+ with the 128gb of unified RAM option.
  
  frame.work/ca/en/products/…/new
  
  I love mine. I’m running NixOS.
  
  source
Mika@piefed.ca ⁨1⁩ ⁨day⁩ ago
If you are capable of running 120b, the mistral cli bigger open source model in agentic mode could be available to you.

How much did your setup cost, if you don’t mind to disclose such information?

source
- Unforeseen@sh.itjust.works ⁨1⁩ ⁨day⁩ ago
  I mentioned my spec above, and I added a 2tb m2, noctua upgrade and a couple expansion modules for the front. I preordered it around 6 months back so before any of the craziness. It was around $3600 CAD
  
  source
  - Mika@piefed.ca ⁨1⁩ ⁨day⁩ ago
    That’s a good value! I was worrying something running mid power LLMs would cost 10k+ today. Sure ram screwup nuked the field, but it’s also not on the top list of my priorities. So it gives me hope that by the time I would be ready it would be in even better shape.
    
    source