Comment on Solve a puzzle for me

• tyler@programming.dev ⁨1⁩ ⁨year⁩ ago

  4o didn’t get it for me.

  Image

  Image

  Image

  Image

  source

  • kautau@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Honestly my answer felt super canned, like someone had asked it before and reported the answer as bad, so that doesn’t surprise me

    source
  • AFKBRBChocolate@lemmy.world ⁨1⁩ ⁨year⁩ ago

    I’m always interested in seeing examples like this where the LLM will get to a right answer after a series of questions (with no additional information) about its earlier wrong responses. I’d love to understand what’s going on in the software that allows the initial wrong answers but gets the eventually right one without an additional input.

    source

    • 31337@sh.itjust.works ⁨1⁩ ⁨year⁩ ago

      One hypothesis is that having more tokens to process lets it “think” longer. Chain of Thought prompting where you ask the LLM to explain its reasoning before giving an answer works similarly. Also, LLMs seem to be better at evaluating solutions that coming up with them, so there is a Tree of Thought technique, where the LLM is asked to generate multiple examples of a reasoning step then pick the “best” reasoning for each reasoning step.

      source