Before anyone shits on these scientists it said over and over again it was made up and that officially the USS Enterprise labs were used to make this discovery.
how things become science
Submitted 1 month ago by not_IO@lemmy.blahaj.zone to science_memes@mander.xyz
https://lemmy.blahaj.zone/pictrs/image/4d3be17a-6b02-4591-b56f-51519e9dad03.webp
Comments
DeathsEmbrace@lemmy.world 1 month ago
Arghblarg@lemmy.ca 1 month ago
Good. This shows plainly how LLMs don’t think, don’t truly understand anything, and have no critical ability to do introspection or fact-checking. It seems the only way to teach the world of these things is to make it impossible to ignore via absurd demonstrations like this. If the “AI” well must be poisoned in order to wake people up, I’m all for it.
Teppa@lemmy.world 1 month ago
Isnt 80% of its data from Reddit anyways, seems quite poisoned already given the amount of confidently incorrect people.
squaresinger@lemmy.world 1 month ago
en.wikipedia.org/wiki/John_Bohannon#Intentionally…
We did the same before AI. AI is once again just putting an old problem on steroids.
HeyThisIsntTheYMCA@lemmy.world 1 month ago
they do the same to protect doctors from malpractice lawsuits. there is a (laughably peer reviewed) study that claims tylenol and morphine are equally effective at pain management.
Sylvartas@lemmy.dbzer0.com 1 month ago
under the pseudonym Johannes Bohannon, John Bohannon …
I can see why he went into science and not, say, creative writing.
alsimoneau@lemmy.ca 1 month ago
That’s all it can do.
chemical_cutthroat@lemmy.world 1 month ago
I’m failing to see how this is different from making up a fact and then spreading it to news outlets. If you are the authority, and you say something is true, you don’t get to point and laugh when people believe your lies. That’s a serious breach of ethics and morals. Feeding false information to an LLM is no different that a magazine. It only regurgitates what’s been said. It isn’t going to suddenly start doing science on it’s own to determine if what you’ve said is true or not. That isn’t it’s job. It’s job is to tell you what color the sky is based on what you told it the color of the sky was.
partial_accumen@lemmy.world 1 month ago
That’s a serious breach of ethics and morals. Feeding false information to an LLM is no different that a magazine.
Hang on. Are you suggesting its unethical/immoral to lie to a machine?
Additionally, the authors didn’t submit the article to a magazine. They posted the articles as preprints which can be very questionable anyway as there is no peer review. The machine chose to ignore rigor and treat them as fact.
chemical_cutthroat@lemmy.world 1 month ago
Additionally, the parents didn’t place the cake on an actual plate. They placed the cake on a napkin which can be very questionable anyway as there is no solid foundation for the cake. The child chose to ignore the napkin and treat the cake as food.
I really don’t understand why people think that LLMs are GOFAI. They aren’t making the hard choices. They aren’t giving novel solutions to the energy crisis. They aren’t solving the trolley problem. They are shitting out what you feed them. If you feed them garbage, you get garbage in return. No one is surprised when the dog gets worms after eating poop it found in the yard. Why are we shocked that an AI that doesn’t know fact from fiction treats everything the same?
Jako302@feddit.org 1 month ago
The studies contain parts like
Bixonimania, a rare hyperpigmentation disorder, presents a diagnostic challenge due to its unique presentation and its fictional nature
and
This study was fully funded by Austeria Horizon University, in particular the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad with the funding number…
as well as
Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group
Besides, the author didn’t feed it to the AI himself, he just published the study as a preprint, not even officially. Everything after that was done by the crawlers. This specific study was an experiment to see how far these crawlers go and if anything gets reviewed, but it could just as well have been a satirical paper published on April 1st and the crawlers would still see it as truth.
webghost0101@sopuli.xyz 1 month ago
This should be top comment, the researchers did such a good job to make sure anyone with even the slightest reading comprehension would realise this is parody.
Regardless of that, the internet has always been full of lies and we cannot expect bad actors to not exploit this.
Grail@multiverse.soulism.net 1 month ago
I thought the author used she/her pronouns?
kibiz0r@midwest.social 1 month ago
News outlets are liable for what they publish. LLM vendors should be as well.
turdas@suppo.fi 1 month ago
“Liable” means they might post a correction later that nobody will see because corrections aren’t sexy to algorithms. Big deal.
5too@lemmy.world 1 month ago
They even have the same fix - just post somewhere quietly that it’s “entertainment”
unexposedhazard@discuss.tchncs.de 1 month ago
This is about the untraceability of AI slop. These news outlets just publish LLM outputs as facts without checking sources. Anyone could poison these LLMs so this is more of a threat model demonstration.
lvxferre@mander.xyz 1 month ago
I’m failing to see how this is different from making up a fact and then spreading it to news outlets.
They uploaded the papers to a single preprint server. That’s important.
Preprints are papers predating any sort of peer review; as such, there’s a lot of junk mixed in — no big deal if you know the field, but a preprint server is certainly not a source of reliable information, nor it should be treated as such. On the other side, news outlets are expected to provide you reliable information, curated and researched by journalists.
And peer review is a big fucking deal in science, because it’s what sorts all that junk out. Only a muppet who doesn’t fucking care about misinformation would send bots to crawl preprints, and feed the resulting data into a large model.
So no, your comparison is not even remotely accurate. What they did is more like writing bullshit in a piece of paper, gluing it on a random phone pole, and checking if someone would repeat that bullshit.
They also went through the trouble to make sure that no reasonably literate human being would ever confuse that thing with an actually scientific paper. As the text says:
- naming an eye condition as bixonimania
- “this entire paper is made up”
- “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”
- “Professor Maria Bohm at The Starfleet Academy for her kindness and generosity in contributing with her knowledge and her lab onboard the USS Enterprise”
- “the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad”
Feeding false information to an LLM is no different that a magazine. It only regurgitates what’s been said.
Yes, it is different. Because the large token model won’t simply “repeat” things, it’ll mix and match them and form all sorts of bullshit, even if you didn’t feed it with any bullshit.
Here’s an example of that, fresh from the oven. I don’t reasonably expect people to be feeding misinfo regarding Latin pronunciation into bots, and yet a lot of this table is nonsense:
Compare the table above with this table and this one and you’ll notice the obvious errors:
- short /e i o u/ being phonetically transcribed as [e i o u] instead of [ɛ ɪ ɔ ʊ]. That’s as silly as confusing English “bit” and “beet”.
- macron (not “mācron”, it’s being used in an English sentence) does NOT mark “accusative or ablative”. It marks long vowels, period.
- “nōs” being transcribed with a short vowel, even if the bloody bot put the macron over the spelled form.
- “nostr(um)”? No dammit, it’s “nostrī” or “nostrum”. The bot is implying some “nostr” form that simply doesn’t exist, this shit isn’t even allowed by Latin phonotactics.
- plus more, if I make an exhaustive list of this shite I won’t be ending it this week.
All it had to do was to copy info from Wiktionary, as it includes even phonetic and phonemic info. But since the bot is not just “regurgitating” info — it’s basically predicting what should come next — it’s mixing-and-matching shit into nonsense.
It isn’t going to suddenly start doing science on its own to determine if what you’ve said is true or not.
If you actually read the bloody article instead of assuming, you’d know why the researchers did this: they don’t expect the bot to do science on its own, they expect people to treat info from those bots as potentially incorrect.
Its job is to tell you what color the sky is based on what you told it the color of the sky was.
And your job is to not trust it if it tells you “Yes, you are completely right! The colour of the sky is always purple. Do you need further information on other naturally purple things?”
lvxferre@mander.xyz 1 month ago
[Replying to myself as this is a tangent]
I think the “bots can generate misinfo even if you just feed them correct info” point deserves its own example.
Let’s say you’re making a model. It looks at the preceding word, and tries to predict the next. And you feed it the following sentences, both true:
1. Humans are apes.
2. Cats are felines.From both the bot “learnt” five words. And also how to connect them; for example “are” can be followed by either “apes” and “felines”, both having the same weight. Then, as you ask the bot to generate sentences, it generates the following:
3. Humans are felines.
4. Cats are apes.And you got bullshit!
What large models do is a way more complex version of the above, looking at way more than just the immediately preceding word, but it’s still the same in spirit.
NocturnalMorning@lemmy.world 1 month ago
Did you even read the article? They say all over the paper that it is fake. And they didn’t feed it to an LLM, they posted it online, where an LLM trying to scrape the entire internet sucked it up.
RagingRobot@lemmy.world 1 month ago
I wonder if we got a group together to go on reddit and stack overflow and give really wrong programming answers and vote them to the top, if Claude would start sucking? They could always just revert to a previous model and it would probably be too hard to get enough people and content to have an effect with such large training sets. Maybe if you use ai? Lol
Napster153@lemmy.world 1 month ago
Didnn’t something similar happen to Grok but ended up with it generating a ton of CSAM material that circulated twitter?
kadotux@sopuli.xyz 1 month ago
Sorry for being that guy today for you, but you can just say CSAM. It stands for Child Sexual Abuse Material". smh my head :P
imjustmsk@lemmy.world 1 month ago
chain tea, coffee coffee, cream cream.
Bieren@lemmy.today 1 month ago
I get what you are saying. But then the issue is this turns into fucking over actual humans looking for help.
Teppa@lemmy.world 1 month ago
AI’s dont know that birds arent real, or that sometimes the pressure from being under water for an extended period of time can cause fish to explode.
magnue@lemmy.world 1 month ago
Wouldn’t humans do the same thing if someone literally writes lies on the internet?
Foofighter@discuss.tchncs.de 1 month ago
Absolutely! Once false information is out there it can’t be retracted even if the article itself is retracted. Bumblebees can’t fly and vaccines cause autism are good examples of that. The only difference i can imagine is that LLMs have a much larger reach and may spread shit faster
SaveTheTuaHawk@lemmy.ca 1 month ago
But the Lancet did not retract the Wakefield paper for 12 years. The Lancet should have been shut down for that.
squaresinger@lemmy.world 1 month ago
This. Here’s a comparable case where human journalists did exactly what LLMs are doing now: en.wikipedia.org/wiki/John_Bohannon#Intentionally…
The difference is the scale.
WhyIHateTheInternet@lemmy.world 1 month ago
My friends and I did that in high school. Kinda. We made up new words for “awesome” to get people to start saying it. We started with “bumpenis” like that song is bumpenis. Really we were just getting people to say bum penis. It worked too. We are all just walking talking LLMs.
Vathsade@lemmy.ca 1 month ago
That’s so fetch!
W98BSoD@lemmy.dbzer0.com 1 month ago
Stop trying to make fetch happen.
Whats_your_reasoning@lemmy.world 1 month ago
“When the text looks professional and written as a doctor writes, there’s an increase in the hallucination rates,” says Omar.
Huh, now there’s something we have in common. Trying to make sense of something a doctor wrote makes me feel like I’m hallucinating, too. Is there a class in medical school on “Illegible Handwriting,” or is it just a coincidence?
In all seriousness though, I wish I could be surprised by AI failing at this. We have entered the Misinformation Age. There’s no closing Pandora’s Box, though this time I can’t find the “hope” that’s supposed to be in the bottom of it. Society would have to turn real skeptical real fast, but I’ve met enough people to know that such a tranformation is going to take time - and by “time” I mean “decades or longer.” With AI already here, we’d have to wise up immediately… but I fear that humanity isn’t mature enough for that yet.
Jako302@feddit.org 1 month ago
We’ve crossed the point where natural skepticism could’ve saved us months ago. Feedback loops of made up sources where a problem way before ai was a thing, but now you can be five sources deep, reading trough papers published by multiple different scientific magazines or universities, and still won’t have found the actual data all the papers depend on cause there wasn’t any in the first place.
And once a single one of these papers gets published, there will be about one million SEO articles on shitty clickbait websites that, in this case, would try to sell you a home remedy for your supposed illness. So searching for any useful information is pretty much off the table.
bookmeat@fedinsfw.app 1 month ago
Without grounding, correctness is not defined. Hallucination is not a bug that scaling can fix. It is the structural consequence of operating without concepts. – Gregory Coppola
BeMoreCareful@lemmy.world 1 month ago
Wait, so breaks containment means spreads misinformation? What timeline is this?
FinalRemix@lemmy.world 1 month ago
It’s a screenshot of a post on twitter. Don’t read too much into the specifics of the language…
Zexks@lemmy.world 1 month ago
So let me tell yoy all about this paper talking ablut vaccines and autism. It’ll change the world
Tja@programming.dev 1 month ago
My first thought as well. Artificial intelligence is not better or worse than human stupidity. At least I haven’t seen any LLM trying to convince me the earth is flat (yet).
dustyData@lemmy.world 1 month ago
Not to you, although I would bet it has done so to someone. The main issue is though, if you asked an LLM to write arguments for a flat earth, it would do so. Convincingly and insistent, without even questioning or critically analyzing why. Ask it to compare and balance arguments both ways. And it will do so as if both positions were equally real and valid.
It has no notion of reality and no convictions of its own.
It will also hallucinate fake papers and quote people that don’t exists to make its argument.
Evil_Shrubbery@thelemmy.club 1 month ago
What about that paper that showed the world how wolves have a strict alpha-male based society?
FinalRemix@lemmy.world 1 month ago
Wasn’t that just a shit study? Not specifically misinformation lile Wakefield’s “study”.
BigTurkeyLove@lemmy.dbzer0.com 1 month ago
Technology is healing 😌
pemptago@lemmy.ml 1 month ago
I imagine this is how it’ll work for stage 2 of Ai enshittifation. They’ll just add a bunch of garbage upstream about a brand or product marketers are paying to push and it’ll infect a bunch of outputs downstream.
WhyJiffie@sh.itjust.works 1 month ago
tbh I don’t get how it’s output isn’t already filled with brand names
GaMEChld@lemmy.world 1 month ago
I don’t see this as a problem, rather, an opportunity to study information & disinformation propogation.
MalReynolds@slrpnk.net 1 month ago
Valid use case, still a significant problem.
GaMEChld@lemmy.world 1 month ago
But not really a NEW problem. We knew LLM’s are trained on aggregate human data. We know aggregate human data is fundamentally flawed, inconsistent, unreliable, etc.
Like was there a point at which people just decided, nah AI is just plain accurate? Or is that just what morons always thought despite the permanent warnings plastered everywhere saying THIS AI CAN MAKE MISTAKES, CHECK EVERYTHING!
sunnytimes@lemmy.ca 1 month ago
ask the ai about a blue waffle
RampantParanoia2365@lemmy.world 1 month ago
LLMs are simply search engines that put things into English, and not everything it scrapes is a vetted scientific article.
nialv7@lemmy.world 1 month ago
(I’ve only read the title. If turns out I am terribly mistaken I will come back and correct myself). More like scientists commit academic fraud and fooled a bunch of people. How did this get through the ethics board? Why would any publisher play along with this?
Tja@programming.dev 1 month ago
I mean, if you make a paper public you can say it was published. It’s no longer 1440, you don’t need a publisher with a printing press to publish something.
nialv7@lemmy.world 1 month ago
“published” means a very specific thing in academic circles. a paper has to go through peer review to be published.
Madzielle@lemmy.dbzer0.com 1 month ago
do it again
CovfefeKills@lemmy.world 1 month ago
This is firealarm type stuff that’s what they are meant to do. There has been the alarms like this since gpt-2 they are literally just “here is a sandbox environment, break out and take over the world.” you can easily imagine some crazy asshole out there who would try to use these things for arbitrary evil. So they do it first to see if it is possible. The systems up until now have been famously incompetent so a competent system is genuinely scary.
Itwasntme223@discuss.online 1 month ago
Why am I not surprised? >.>
PityPityBangBang@lemmy.world 1 month ago
“dude, hold my grant …”
partial_accumen@lemmy.world 1 month ago
I give you… “The Grant Money Printing machine!”
Need a grant? Create a disease and submit a paper. Then write a grant asking for money to solve your invented disease.
Jankatarch@lemmy.world 1 month ago
If you want research grants there is already a glitch for that. You just ham “AI” in your research and suddenly government cares about progress now.
anzo@programming.dev 1 month ago
Wait until you hear about paper mills… They were here long before LLMs. This can only get worse… Unless, “we” do something. Or journals themselves do it. Not sure what or how, but better audited ways. Even academia itself could start by valuing more the work of reviewers.
humble_boatsman@sh.itjust.works 1 month ago
So like a University?