Comment on The Open-Source Software Saving the Internet From AI Bot Scrapers
who@feddit.org 1 day agoInteresting. Judging by that option’s name, it seems to refer to use of the HTML <meta> tag to refresh a page.
developer.mozilla.org/en-US/docs/…/http-equiv
Neither this tag nor using it for refresh is new at all. I don’t think I’ve seen it used to detect bots, though. I wonder what Anubis is doing here.
JohnEdwa@sopuli.xyz 1 day ago
It’s simply checking if the connection is from an actual browser, as a scraper pretending to be one won’t actually refresh the page as instructed. It’s going to buy some time, but like the rest of Anubis in general, it will only work until the scrapers get modified to work around it.