You’re missing a crucial detail- the discussions of LLMs as just being probability machines ignores the fact that when we’re talking about “what is the most likely next word?” The answer to that question isn’t merely just “What, in the training corpus, was the most likely word to follow in this instance?” But rather there is a “finger on the scale” so to speak in favor of certain types of responses, and this is frequently updated. You cannot have a useful LLM without this because it will talk as if it is a human with a sense of self, display blatant prejudice just because it’s common, and say creepy things (see the Microsoft “Sydney” fiasco) because the humans who wrote the sources in the training corpus do have a sense of self, do hold prejudices, and express thoughts and feelings that are inappropriate coming from a chatbot. When done intentionally and carefully, this creates a much more useful product, but when done poorly it potentially makes things worse. It seems that at least part of that weighting is based on user interaction, which is what I was talking about with models getting dumber the more they interact with the general public.

Furthermore, the newest versions of ChatGPT attempt to include “reasoning” as an actual feature of the response. I’ve played around with them and they are definitely a lot better at logic and math problems than older models but not necessarily less prone to “hallucinations” when it comes to factual information. I haven’t read a whole lot about how the “reasoning” works because I have been a lot more interested in non-LLM methods lately but it is intended to combat the issue you described. Personally I am not convinced this will fix much of anything in its current strategy but it’s certainly interesting to see.