Comment on A big announcement from Skill Up (new website)
HeyJoe@lemmy.world 11 months agoIs there any true way to block it? Does the crawler literally use the same access (443) as us to scrape content? If so, the only other thing I can think of is to block all known IP’s that AI crap originates from, but that sounds daunting and impossible to catch everything.
pirateKaiser@sh.itjust.works 11 months ago
There’s no fullproof way. Even if you somehow block every crawling automation, there’s still puppeteering where the bot behaves just like a normal user.