Comment on OpenAI releases a free GPT model that can run on your laptop

nathan@piefed.alphapuggle.dev ⁨5⁩ ⁨days⁩ ago

• sefra1@lemmy.zip ⁨5⁩ ⁨days⁩ ago

  Isn’t that true for most models until someone destiles and quantises them so they can run on common hardware?

  source

  • fuckwit_mcbumcrumble@lemmy.dbzer0.com ⁨5⁩ ⁨days⁩ ago

    This is the internet, we’re only allowed to be snarky here.

    source
  • CyberSeeker@discuss.tchncs.de ⁨4⁩ ⁨days⁩ ago

    Yes, but 20 billion parameters is too much for most GPUs, regardless of quantization. You would need at least 14GB, and even that’s unlikely without offloading major parts to the CPU and system RAM (which kills the token rate).

    source

    • fuckwit_mcbumcrumble@lemmy.dbzer0.com ⁨3⁩ ⁨days⁩ ago

      I tried it out last night and it ran quite well on my heavily thermally limited i9 11950h/rtx 3080 laptop. I had maybe 6 or 7 gigs of main ram used in total, with docker running. It was only using about 12 gigs of vram in my very limited testing.

      source
  • Ghoelian@lemmy.dbzer0.com ⁨5⁩ ⁨days⁩ ago

    I mean yeah, but that doesn’t make the title any lore true.

    source