Comment on Enshittification of ChatGPT
Zaleramancer@beehaw.org 4 weeks agoHello again! So, I am interested in engaging with this question, but I have to say: My initial post is about how an LLM cannot provide actual, real citations with any degree of academic rigor for a random esoteric topic. This is because it cannot understand what a citation is, only what it is shaped like.
An LLM deals with context over content. They create structures that are legible to humans, and they are quite good at that. An LLM can totally create an entire conversation with a fictional character in their style and voice- that doesn’t mean it knows what that character is. Consider how AI art can have problems that arise from the fact that they understand the shape of something, but they don’t know what it actually is- that’s why early AI art had a lot of problems with objects ambigiously becoming other objects. The fidelity of these creations has improved with the technology, but that doesn’t imply understanding of the content.
Do you think an LLM understands the idea of truth? Do you think if you ask it to say a truthful thing, and be very sure of itself and think it over, it will produce something that’s actually more accurate or truthful- or just something that has the language hall-marks of being truthful? I know that an LLM will produce complete fabrications that distort the truth if you expect a base-line level of rigor from them, and I proved that above, in that the LLM couldn’t even accurately report the name of a book it was supposedly using as a source.
What is understanding, if the LLM can make up an entire author, book and bibliography if you ask it to tell you about the real world?
localhost@beehaw.org 4 weeks ago
Hey again! First of all, thank you for continuing to engage with me in good faith and for your detailed answers. We may differ in our opinions on the topic but I’m glad that we are able to have a constructive and friendly discussion nonetheless :)
I agree with you that LLMs are bad at providing citations. Similarly they are bad at providing urls, id numbers, titles, and many other things that require high accuracy memorization. I don’t necessarily agree that this is a definite proof of their incapability to understand.
In my view, LLMs are always in an “exam mode”. That is to say, due to the way they are trained, they have to provide answers even if they don’t know them. This is similar to how students act when they are taking an exam - they make up facts not because they’re incapable of understanding the question, but because it’s more beneficial for them to provide a partially wrong answer than no answer at all.
I’m also not taking a definitive position on whether or not LLMs have capability to understand (IMO that’s pure semantics). I am pushing back against the recently widespread idea that they provably don’t. I think LLMs have some tasks that they are very capable at and some that they are not. It’s disingenuous and possibly even dangerous to downplay a powerful technology under a pretense that it doesn’t fit some very narrow and subjective definition of a word.
And this is unfortunately what I often see here, on other lemmy instances, and on reddit - people not only redefining what “understand”, “reason”, or “think” means so that generative AI falls outside of it, but then using this self-proclaimed classification to argue that they aren’t capable of something else entirely. A car doesn’t lose its ability to move if I classify it as a type of chair. A bomb doesn’t stop being dangerous if I redefine what it means to explode.
I don’t think it’s impossible. You can give ChatGPT a true statement, instruct it to lie to you about it, and it will do it. You can then ask it to point out which part of its statement was a lie, and it will do it. You can interrogate it in numerous ways that don’t require exact memorization of niche subjects and it will generally produce an output that, to me, is consistent with the idea that it understands what truth is.
Let me also ask you a counter question: do you think a flat-earther understands the idea of truth? After all, they will blatantly hallucinate incorrect information about the Earth’s shape and related topics. They might even tell you internally inconsistent statements or change their mind upon further questioning. And yet I don’t think this proves that they have no understanding about what truth is, they just don’t recognize some facts as true.
Zaleramancer@beehaw.org 4 weeks ago
Hi, once more, I’m happy to have a discussion about this. I have very firm views on it, and enjoy getting a chance to discuss them and work towards an ever greater understanding of the world.
I completely understand the desire to push back against certain kinds of “understandings” people have about LLM due to their potentially harmful inaccuracy and the misunderstandings that they could create. I have had to deal with very weird, like, existentialist takes on AI art lacking a quintessential humanity that all human art is magically endowed with- which, come on, there are very detailed technical art reasons why they’re different, visually! It’s a very complicated phenomenon, but, it’s not an inexplicable cosmic mystery! Take an art critique class!
Anyway, I get it- I have appreciated your obvious desire to have a discussion.
On the subject of understanding, I guess what I mean is this: Based on everything I know about an LLM, their “information processing” happens primarily in their training. This is why you can run an LLM instance on, like, a laptop but it takes data centers to train them. They do not actually process new information, because if they did, you wouldn’t need to train them, would you- you’d just have them learn and grow over time. An LLM breaks its training data down into patterns and shapes and forms, and uses very advanced techniques to generate the most likely continuation of a collection of words. You’re right in that they must answer, but that’s because their training data is filled with that pattern of answering the question. The natural continuation of a question is, always, an answer-shaped thing. Because of the miracles of science, we can get a very accurate and high fidelity simulation of what that answer would look like!
Understanding, to me, implies a real processing of new information and a synthesis of prior and new knowledge to create a concept. I don’t think it’s impossible for us to achieve this, technologically, humans manage it and I’m positive that we could eventually figure out a synthetic method of replicating it. I do not think an LLM does this. The behavior they exhibit and the methods they use seem radically inconsistent with that end. Because, the ultimate goal of them was not to create a thinking thing, but to create something that’s able to make human-like speech that’s coherent, reliable and conversational. They totally did that! It’s incredibly good at that. If it were not for the context of them politically, environmentally and economically, I would be so psyched about using them! I would have been trying to create templates to get an LLM to be an amazing TTRPG oracle if it weren’t for the horrors of the world.
It’s incredible that we were able to have a synthetic method of doing that! I just wish it was being used responsibly.
An LLM, based on how it works, cannot understand what it is saying, or what you are saying, or what anything means. It can continue text in a conversational and coherent way, with a lot of reliability on how it does that. The size, depth and careful curation of its training data mean that those responses are probably as accurate to being an appropriate response as they can be. This is why, for questions of common knowledge, or anything you’d do a light google for, they’re fine. They will provide you with an appropriate response because the question probably exists hundreds of thousands of times in the training data; and, the information you are looking for also exists in huge redundancies across the internet that got poured into that data. If I ask an LLM which of the characters of My Little Pony has a southern accent, they will probably answer correctly because that information has been repeated so much online that it probably dwarfs the human written record of all things from 1400 and earlier.
The problem becomes evident when you ask something that is absolutely part of a structured system in the english language, but which has a highly variable element to it. This is why I use the “citation problem” when discussing them, because they’re perfect for this: A citation is part of a formal or informal essay, which are deeply structured and information dense, making them great subjects for training data. Their structure includes a series of regular, repeating elements in particular orders: Name, date, book name, year, etc- these are present and repeated with such regularity that the pattern must be quite established for the LLM as a correct form of speech. The names of academic books are often also highly patterned, and an LLM is great at creating human names, so there’s no problem there.
The issue is this: How can an LLM tell if a citation it makes is real? It gets a pattern that says, “The citation for this information is:” and it continues that pattern by putting a name, date, book title, etc in that slot. However, this isn’t like asking what a rabbit is- the pattern of citations leads into an endless warren of hundreds of thousands names, book titles, dates, and publishing companies. It generates them, but it cannot understand what a citation really means, just that there is a pattern it must continue- so it does.
A flat-earther has some understanding of what truth is, even if their definition is divergent from the norm. The things they say are deeply inaccurate, but you can tell that they are the result of a chain of logic that you can ask about and follow. It’s possible to trace flat-earth ideas down to sources. They’re incorrect, but they’re arrived at because of an understanding of prior (incorrect) information. A flat-earther does not always invent their entire argument and the basis for their beliefs on the spot, they are presenting things they know about from prior events- they can show the links. An LLM cannot tell you how it arrived at a conclusion, because if you ask it, you are just receiving a new continuation of your prior text. Whatever it says is accurate only when probability and data set size is on its side.
localhost@beehaw.org 3 weeks ago
This is partially true and partially not. It’s true that LLMs can’t learn anything wildly novel, because they are not flexible enough for this. But they can process new information, in fact they do it all the time. You can produce conversations that no one had before, and yet LLMs like ChatGPT will respond to it appropriately. This is more than just shape matching.
In fact, there are techniques like Few-Shot Learning and Chain of Thought that rely on the LLMs’ ability to learn from context and revise its own answers.
IMO citation problem is not testing capability to understand. It’s testing knowledge, memorization, and ability to rate its own confidence. Keep in mind that ChatGPT and most other LLMs will tell you when they perform web searches - if they don’t then they’re likely working off context alone. Enabling web search would greatly increase the accuracy of LLM’s answers.
Unlike LLMs we have somewhat robust ability to rate how confident we are about our recollections, but even in humans memory can be unreliable and fail silently. I’ve had plenty of conversations where I argue with someone about something that one of us remembers happening and the other one is certain didn’t happen - or happened differently. Without lies or misunderstandings, two people who had at some point memorized the same thing can later on confidently disagree on the details. Human brains are not databases and they will occasionally mangle memories or invent concepts that don’t exist.
And even that is completely skipping over people with mental disorders that affect their thinking patterns. Is someone with psychosis incapable of understanding anything because they hold firm beliefs on things that cannot be traced to any source? Are people with frontal lobe damage who develop intense confabulations incapable of understanding? How about compulsive liars? Are you willing to label a person or an entire demographic as incapable of understanding if they fail your citation test?
There are techniques like Chain of Thought that make LLMs think before generating response. Those systems will be able to tell you how they arrived at the conclusion.
But humans are also fairly prone to rationalization after the fact. There was a famous experiment on people who had to have functional hemispherectomy for medical reasons, where the left hemisphere makes up an explanation for right hemisphere’s choices despite not knowing the true reason:
“Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice. We then asked the left hemisphere, the only one that can talk, why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism.”
Zaleramancer@beehaw.org 2 weeks ago
I’ve really enjoyed this discussion, but I haven’t been able to respond because I don’t have the mental bandwidth right now. Thanks for being such a good conversational partner and I think you make some very interesting points that helped me develop my own opinions.