Comment on A big announcement from Skill Up (new website)
HeyJoe@lemmy.world 1 day agoIs there any true way to block it? Does the crawler literally use the same access (443) as us to scrape content? If so, the only other thing I can think of is to block all known IP’s that AI crap originates from, but that sounds daunting and impossible to catch everything.
pirateKaiser@sh.itjust.works 14 hours ago
There’s no fullproof way. Even if you somehow block every crawling automation, there’s still puppeteering where the bot behaves just like a normal user.