Comment on The Open-Source Software Saving the Internet From AI Bot Scrapers
exu@feditown.com 9 hours agoThere’s an option using some very new HTML tag, but it’s not the default.
Comment on The Open-Source Software Saving the Internet From AI Bot Scrapers
exu@feditown.com 9 hours agoThere’s an option using some very new HTML tag, but it’s not the default.
who@feddit.org 6 hours ago
Interesting. Judging by that option’s name, it seems to refer to use of the HTML
<meta>tag to refresh a page.developer.mozilla.org/en-US/docs/…/http-equiv
Neither this tag nor using it for refresh is new at all. I don’t think I’ve seen it used to detect bots, though. I wonder what Anubis is doing here.
JohnEdwa@sopuli.xyz 26 minutes ago
It’s simply checking if the connection is from an actual browser, as a scraper pretending to be one won’t actually refresh the page as instructed. It’s going to buy some time, but like the rest of Anubis in general, it will only work until the scrapers get modified to work around it.