Depends on the product. From an original AI research point of view this is what you want, a model that can realize it is missing information and deviates from giving a result. But once profit became involved, marketing requires a fully confident output to get everyone to buy in. So we get what we get and not something that's more reliable.
Comment on Why OpenAI’s solution to AI hallucinations would kill ChatGPT tomorrow
Dave@lemmy.nz 5 days ago
Consider the implications if ChatGPT started saying “I don’t know” to even 30% of queries – a conservative estimate based on the paper’s analysis of factual uncertainty in training data. Users accustomed to receiving confident answers to virtually any question would likely abandon such systems rapidly.
I think we would just be more careful with how we used the technology. E.g. don’t autocomplete code if the threshold is not met for reasonable certainty.
I would argue that it’s more useful having a system that says it doesn’t know half the time than a system that’s confidently wrong half the time
Rhaedas@fedia.io 5 days ago
WalnutLum@lemmy.ml 5 days ago
It’s not just that, it’s also the fact they scored the responses based on user feedback, and users tend to give better feedback for more confident, even if wrong, responses.
30p87@feddit.org 5 days ago
Obviously. But more useful ≠ more money. So the fascocapitalists will ofc not implement that.