Comment on Wikipedia is giving AI developers its data to fend off bot scrapers

<- View Parent
p03locke@lemmy.dbzer0.com ⁨6⁩ ⁨days⁩ ago

It’s a common problem. People writing bot scrapers for public data, which costs a lot of bandwidth, when they could have easily just downloaded the entire dataset from a dedicated link. Finding better ways to tell them “Hey, morons, go download the goddamn link!” saves on that bandwidth and web server CPU.

source
Sort:hotnewtop