Comment

Comment on AI hallucinations are impossible to eradicate — but a recent, embarrassing malfunction from one of China’s biggest tech firms shows how they can be much more damaging there than in other countries

lvxferre@mander.xyz ⁨2⁩ ⁨months⁩ ago

This article shows rather well three reasons why I don’t like the term “hallucination”, when it comes to LLM output.

It’s a catch-all term that describes neither the nature nor the gravity of the problematic output. Failure to address the prompt? False output, fake info? Immoral and/or harmful output? Pasting verbatim training data? Output that is supposed to be moderated against? It’s all “hallucination”.
It implies that, under the hood, the LLM is “malfunctioning”. It is not - it’s doing what it is supposed to do, to chain tokens through weighted probabilities. Contrariwise to the tech bros’ wishful belief, LLMs do not pick words based on the truth value or morality of the output. That’s why hallucinations won’t go away, at least not for the current architecture of text generators.
It lumps together those incorrect outputs with what humans would generate on situations of poor reasoning. This “it works like a human” metaphor obscures what happens, instead of clarifying it.

On the main topic of the article. Are LLMs useful? Sure! I use them myself. However only a fool would try to shove LLMs everywhere, with no regards to how intrinsically [yes] unsafe they are. And yet it’s what big tech is doing, regardless of being Chinese or United-Statian or Russian or German or whatever.

source

Sort:hotnew top

HobbitFoot@thelemmy.club ⁨2⁩ ⁨months⁩ ago
I feel like “hallucination” was chosen as the word because of what it implies.

It doesn’t imply a bad algorithm, which makes the company look bad since hallucinations are out of a person’s control. It doesn’t imply using poor training data for the same reason.

But hallucination also masks the development of the model. A small kid might say something racist based on what they grew up with, but we would likely call that child immature. Same if an AI doesn’t fully understand a question or repeats a wrong answer that was given to it by someone as a laugh.

No one is fault. It was just a hallucination.

source
AndrasKrigare@beehaw.org ⁨2⁩ ⁨months⁩ ago

It implies that, under the hood, the LLM is “malfunctioning”. It is not - it’s doing what it is supposed to do, to chain tokens through weighted probabilities.

I don’t really agree with that argument. By that logic, there’s really no such thing as a software bug, since the software is always doing what it’s supposed to be doing: giving predefined instructions to a processor that performs some action. It’s “supposed to” provide a useful response to prompts, anything other than is it not what it should be and could be fairly called a malfunction.

source
- lvxferre@mander.xyz ⁨2⁩ ⁨months⁩ ago
  When it comes to the code itself you’re right, there’s no difference between “bug” and “not a bug”. The difference is how humans classify the behaviour.
  
  And yet there’s a clear mismatch between what the developers of those large “language” models know that they’re able to do, versus what LLMs are being promoted for, and that difference is what is being called “hallucination”. They are not intelligent systems, the info that they output is not reliably accurate, it’s often useless rubbish. But instead of acknowledging it they label it “hallucination”.
  
  Perhaps an example would be good here. Suppose that I made a text editor; it works nicely as a text editor and nothing much else. Then I make it automatically find and replace the string “=2+2” with “4”, and use it to showcase my text editor as if it was a calculator. “Look, it can do maths!”.
  
  Then the user types down “=3+3”, expecting the “spreadsheet” to output “6”, and it doesn’t. Can we really claim that the user found a “bug”? Not really. It’s just that I’m a phony and I sold him a text editor as if it was a calculator.
  
  And yet that’s exactly what happens with LLMs.
  
  source
  - AndrasKrigare@beehaw.org ⁨1⁩ ⁨month⁩ ago
    I think to some extent it’s a matter of scale, though. If I advertise something as a calculator capable of doing all math, and it can only do one problem, it is so drastically far away from its intended purpose that the meaning kinda breaks down. I don’t think it would be wrong to say “it malfunctions in 99.999999% of use cases” but it would be easier to say that it just doesn’t work.
    
    Continuing (and torturing) that analogy, if we did the disgusting work of precomputing all 2 number math problems for integers from -1,000,000 to 1,000,000 and I think you could say you had a (really shitty and slow) calculator, which “malfunctions” for numbers outside that range if you don’t specify the limitation ahead of time. Not crazy different from software which has issues with max_int or small buffers.
    
    If it were the case that there had only been one case of a hallucination with LLMs, I think we could pretty safely call that a malfunction (and we wouldn’t be having this conversation). If it happens 0.000001% of the time, I think we could still call it a malfunction and that it performs better than a lot of software. 99.999% of the time, it’d be better to say that it just doesn’t work. I don’t think there is, or even needs to be, some unified understanding of where the line is between them.
    
    Really my point is there are enough things to criticize about LLMs and people’s use of them, this seems like a really silly one to try and push.
    
    source
    lvxferre@mander.xyz ⁨1⁩ ⁨month⁩ ago
    
    Really my point is there are enough things to criticize about LLMs and people’s use of them, this seems like a really silly one to try and push.
    
    The comment that you’re replying to is fairly specifically criticising the usage of the word “hallucination” to misrepresent the nature of the undesirable LLM output, in the context of people selling you stuff by what it is not.
    
    It is not “pushing” another “thing to criticise about LLMs”. OK? I have my fair share of criticism against LLMs themselves, but that is not what I’m doing right now.
    
    Continuing (and torturing) that analogy, […] max_int or small buffers.
    
    When we extend analogies they often break in the process. That’s the case here.
    
    Originally the analogy works because it shows a phony selling a product by what it is not. By making the phony to precompute 4*10¹² equations (a completely unrealistic situation), he stops being a phony to become a muppet doing things the hard way.
    
    If it were the case that there had only been one case of a hallucination with LLMs, I think we could pretty safely call that a malfunction
    
    If it happens 0.000001% of the time, I think we could still call it a malfunction and that it performs better than a lot of software.
    
    Emphases mine. Those “ifs” represent a completely unrealistic situation, that does not show anything useful about the real situation.
    
    We know that LLMs output “hallucinations” way more than just once, or 0.000001% of the time. They’re common enough to show you how LLMs work.
    
    source
- t3rmit3@beehaw.org ⁨2⁩ ⁨months⁩ ago
  Except Lvxferre is actually correct; LLMs are not capable of determining what is useful or not useful, nor can they ever be as a fundamental part of their models; they are simply strings of weighted tokens/numbers. The LLM does not “know” anything, it is approximating text similar to what it was trained on.
  
  It would be like training a parrot and then being upset that it doesn’t understand what the words mean.
  
  The only way to ensure they produce only useful output is to screen their answers against a known-good database of information, at which point you don’t need the AI model anyways.
  
  source
  - AndrasKrigare@beehaw.org ⁨2⁩ ⁨months⁩ ago
    Where I don’t think your argument fits is that it could be applied to things LLMs can currently do. If I have an insufficiently trained model which produces a word salad to every prompt, one could say “that’s not a malfunction, it’s still applying weights.”
    
    The malfunction is in having a system that produces useful results. An LLM is just the means for achieving that result, and you could argue it’s the wrong tool for the job and that’s fine. If I put gasoline in my diesel car and the engine dies, I can still say the car is malfunctioning. It’s my fault, and the engine wasn’t ever supposed to have gas in it, but the car is now “failing to function in a normal or satisfactory manner,” the definition of malfunction.
    
    source
    t3rmit3@beehaw.org ⁨2⁩ ⁨months⁩ ago
    The purpose of an LLM, at a fundamental level, is to approximate text it was trained on. If it was trained on gibberish, outputting gibberish wouldn’t be a bug. If it wasn’t, outputting gibberish would be indicative of a bug.
    
    I can still say the car is malfunctioning.
    
    A better analogy would be selling someone a diesel car, when they wanted an electric vehicle, and them being upset when it requires refueling with gas. The car is t malfunctioning in that case, the salesman was.
    
    source
    -> View More Comments
- technocrit@lemmy.dbzer0.com ⁨1⁩ ⁨month⁩ ago
  
  It’s “supposed to” provide a useful response to prompts
  
  There’s no objective definition of “useful”. Objectively the program is working. Subjectively it’s not working how certain people want it to work.
  
  source
  - AndrasKrigare@beehaw.org ⁨1⁩ ⁨month⁩ ago
    We’re talking about the meaning of “malfunction” here, we don’t need to overthink it and construct a rigorous proof or anything. The creator of the thing can decide what the thing they’re creating is supposed to do. You can say
    
    hey, it did X, was that supposed to happen?
    
    no, it was not supposed to do that, that’s a malfunction.
    
    We don’t need to go to
    
    Actually you never sufficiently defined its function to cover all cases in an objective manner, so ACTUALLY it’s not a malfunction!
    
    Whatever, it still wasn’t supposed to do that
    
    source
Kissaki@beehaw.org ⁨2⁩ ⁨months⁩ ago
I wouldn’t call pasting verbatim training data hallucination when it fits the prompt.

I feel like you’re unfittingly mixing tool target behavior with technical limitations. Yes, it’s not knowingly reasoning. But that doesn’t change that the user interface is a prompt-style, with the goal of answering.

I think it’s fitting terminology for encompassing multiple issues of false answers.

How would you call it? Only by their specific issues? Or would you use a general term, like “error” or “wrong”?

source
- lvxferre@mander.xyz ⁨2⁩ ⁨months⁩ ago
  
  I wouldn’t call pasting verbatim training data hallucination when it fits the prompt. It’s not necessarily making stuff up.
  
  I’ve seen it being called hallucination plenty of times. Because the output is undesirable - even if it satisfies the prompt, it is not something you’d want the end user to see, as it shows that the whole thing is built upon the unpaid labour of everyone who uses the internet.
  
  source